[jira] [Commented] (HBASE-17241) Avoid compacting already compacted mob files with _del files
[ https://issues.apache.org/jira/browse/HBASE-17241?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15726559#comment-15726559 ] huaxiang sun commented on HBASE-17241: -- Thanks Ted and Anoop for review! > Avoid compacting already compacted mob files with _del files > - > > Key: HBASE-17241 > URL: https://issues.apache.org/jira/browse/HBASE-17241 > Project: HBase > Issue Type: Improvement > Components: mob >Affects Versions: 2.0.0 >Reporter: huaxiang sun >Assignee: huaxiang sun > Fix For: 2.0.0 > > Attachments: HBASE-17241-master-002.patch, > HBASE-17241-master-003.patch, HBASE-17241.master.001.patch > > > Today if there is only one file in the partition, and there is no _del files, > the file is skipped. With del file, the current logic is to compact the > already-compacted file with _del file. Let's say there is one mob file > regionA20161101***, which was compacted. On 12/1/2016, there is _del file > regionB20161201**_del, mob compaction kicks in, regionA20161101*** is less > than the threshold, and it is picked for compaction. Since there is a _del > file, regionA20161101 and regionB20161201***_del are compacted into > regionA20161101**_1 . After that, regionB20161201**_del cannot be deleted > since it is not a allFile compaction. The next mob compaction, > regionA20161101**_1 and regionB20161201**_del will be picked up again and be > compacted into regionA20161101***_2. So in this case, it will cause more > unnecessary IOs. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-17241) Avoid compacting already compacted mob files with _del files
[ https://issues.apache.org/jira/browse/HBASE-17241?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15722533#comment-15722533 ] huaxiang sun commented on HBASE-17241: -- Hi [~anoop.hbase], per my understanding, this is the current logic when all files's size is larger than the threshold. In compactMobFiles(), if the partitions is empty, it will skip the rest of logic and return. Yeah, this can optimized a bit. > Avoid compacting already compacted mob files with _del files > - > > Key: HBASE-17241 > URL: https://issues.apache.org/jira/browse/HBASE-17241 > Project: HBase > Issue Type: Improvement > Components: mob >Affects Versions: 2.0.0 >Reporter: huaxiang sun >Assignee: huaxiang sun > Fix For: 2.0.0 > > Attachments: HBASE-17241-master-002.patch, > HBASE-17241-master-003.patch, HBASE-17241.master.001.patch > > > Today if there is only one file in the partition, and there is no _del files, > the file is skipped. With del file, the current logic is to compact the > already-compacted file with _del file. Let's say there is one mob file > regionA20161101***, which was compacted. On 12/1/2016, there is _del file > regionB20161201**_del, mob compaction kicks in, regionA20161101*** is less > than the threshold, and it is picked for compaction. Since there is a _del > file, regionA20161101 and regionB20161201***_del are compacted into > regionA20161101**_1 . After that, regionB20161201**_del cannot be deleted > since it is not a allFile compaction. The next mob compaction, > regionA20161101**_1 and regionB20161201**_del will be picked up again and be > compacted into regionA20161101***_2. So in this case, it will cause more > unnecessary IOs. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-17241) Avoid compacting already compacted mob files with _del files
[ https://issues.apache.org/jira/browse/HBASE-17241?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15721564#comment-15721564 ] Anoop Sam John commented on HBASE-17241: So some file paths are removed as per the new logic Still we continue with below with out any check PartitionedMobCompactionRequest request = new PartitionedMobCompactionRequest( filesToCompact.values(), allDelFiles); When filesToCompact become empty, still to go ahead this way? > Avoid compacting already compacted mob files with _del files > - > > Key: HBASE-17241 > URL: https://issues.apache.org/jira/browse/HBASE-17241 > Project: HBase > Issue Type: Improvement > Components: mob >Affects Versions: 2.0.0 >Reporter: huaxiang sun >Assignee: huaxiang sun > Fix For: 2.0.0 > > Attachments: HBASE-17241-master-002.patch, > HBASE-17241-master-003.patch, HBASE-17241.master.001.patch > > > Today if there is only one file in the partition, and there is no _del files, > the file is skipped. With del file, the current logic is to compact the > already-compacted file with _del file. Let's say there is one mob file > regionA20161101***, which was compacted. On 12/1/2016, there is _del file > regionB20161201**_del, mob compaction kicks in, regionA20161101*** is less > than the threshold, and it is picked for compaction. Since there is a _del > file, regionA20161101 and regionB20161201***_del are compacted into > regionA20161101**_1 . After that, regionB20161201**_del cannot be deleted > since it is not a allFile compaction. The next mob compaction, > regionA20161101**_1 and regionB20161201**_del will be picked up again and be > compacted into regionA20161101***_2. So in this case, it will cause more > unnecessary IOs. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-17241) Avoid compacting already compacted mob files with _del files
[ https://issues.apache.org/jira/browse/HBASE-17241?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15721559#comment-15721559 ] Hudson commented on HBASE-17241: SUCCESS: Integrated in Jenkins build HBase-Trunk_matrix #2075 (See [https://builds.apache.org/job/HBase-Trunk_matrix/2075/]) HBASE-17241 Avoid compacting already compacted mob files with _del files (tedyu: rev 0d6b872d9dca4c040a172eed7b0153ece36c0a0d) * (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/mob/compactions/TestPartitionedMobCompactor.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/mob/compactions/PartitionedMobCompactor.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/mob/compactions/PartitionedMobCompactionRequest.java > Avoid compacting already compacted mob files with _del files > - > > Key: HBASE-17241 > URL: https://issues.apache.org/jira/browse/HBASE-17241 > Project: HBase > Issue Type: Improvement > Components: mob >Affects Versions: 2.0.0 >Reporter: huaxiang sun >Assignee: huaxiang sun > Fix For: 2.0.0 > > Attachments: HBASE-17241-master-002.patch, > HBASE-17241-master-003.patch, HBASE-17241.master.001.patch > > > Today if there is only one file in the partition, and there is no _del files, > the file is skipped. With del file, the current logic is to compact the > already-compacted file with _del file. Let's say there is one mob file > regionA20161101***, which was compacted. On 12/1/2016, there is _del file > regionB20161201**_del, mob compaction kicks in, regionA20161101*** is less > than the threshold, and it is picked for compaction. Since there is a _del > file, regionA20161101 and regionB20161201***_del are compacted into > regionA20161101**_1 . After that, regionB20161201**_del cannot be deleted > since it is not a allFile compaction. The next mob compaction, > regionA20161101**_1 and regionB20161201**_del will be picked up again and be > compacted into regionA20161101***_2. So in this case, it will cause more > unnecessary IOs. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-17241) Avoid compacting already compacted mob files with _del files
[ https://issues.apache.org/jira/browse/HBASE-17241?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15721158#comment-15721158 ] Ted Yu commented on HBASE-17241: Test failure was not related to the patch. > Avoid compacting already compacted mob files with _del files > - > > Key: HBASE-17241 > URL: https://issues.apache.org/jira/browse/HBASE-17241 > Project: HBase > Issue Type: Improvement > Components: mob >Affects Versions: 2.0.0 >Reporter: huaxiang sun >Assignee: huaxiang sun > Attachments: HBASE-17241-master-002.patch, > HBASE-17241-master-003.patch, HBASE-17241.master.001.patch > > > Today if there is only one file in the partition, and there is no _del files, > the file is skipped. With del file, the current logic is to compact the > already-compacted file with _del file. Let's say there is one mob file > regionA20161101***, which was compacted. On 12/1/2016, there is _del file > regionB20161201**_del, mob compaction kicks in, regionA20161101*** is less > than the threshold, and it is picked for compaction. Since there is a _del > file, regionA20161101 and regionB20161201***_del are compacted into > regionA20161101**_1 . After that, regionB20161201**_del cannot be deleted > since it is not a allFile compaction. The next mob compaction, > regionA20161101**_1 and regionB20161201**_del will be picked up again and be > compacted into regionA20161101***_2. So in this case, it will cause more > unnecessary IOs. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-17241) Avoid compacting already compacted mob files with _del files
[
https://issues.apache.org/jira/browse/HBASE-17241?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15720865#comment-15720865
]
Hadoop QA commented on HBASE-17241:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 16s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
48s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 41s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
46s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
14s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
58s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 32s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
51s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 43s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 43s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
48s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
15s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
28m 38s {color} | {color:green} Patch does not cause any errors with Hadoop
2.6.1 2.6.2 2.6.3 2.6.4 2.6.5 2.7.1 2.7.2 2.7.3 or 3.0.0-alpha1. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m 1s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 29s
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 92m 5s {color}
| {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
14s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 133m 39s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hbase.regionserver.TestHRegion |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.12.3 Server=1.12.3 Image:yetus/hbase:8d52d23 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12841674/HBASE-17241-master-003.patch
|
| JIRA Issue | HBASE-17241 |
| Optional Tests | asflicense javac javadoc unit findbugs hadoopcheck
hbaseanti checkstyle compile |
| uname | Linux d7827931f0f0 3.13.0-93-generic #140-Ubuntu SMP Mon Jul 18
21:21:05 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
|
| git revision | master / c73180d |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| unit |
https://builds.apache.org/job/PreCommit-HBASE-Build/4780/artifact/patchprocess/patch-unit-hbase-server.txt
|
| unit test logs |
https://builds.apache.org/job/PreCommit-HBASE-Build/4780/artifact/patchprocess/patch-unit-hbase-server.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HBASE-Build/4780/testReport/ |
| modules | C: hbase-server U: hbase-server |
| Console output |
https://builds.apache.org/job/PreCommit-HBASE-Build/4780/console |
| Powered by | Apache Yetus 0.3.0 http://yetus.apache.org |
This message was automatically generated.
> Avoid compacting already compacted mob files with _del files
> -
>
>
[jira] [Commented] (HBASE-17241) Avoid compacting already compacted mob files with _del files
[ https://issues.apache.org/jira/browse/HBASE-17241?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15720347#comment-15720347 ] Ted Yu commented on HBASE-17241: +1 > Avoid compacting already compacted mob files with _del files > - > > Key: HBASE-17241 > URL: https://issues.apache.org/jira/browse/HBASE-17241 > Project: HBase > Issue Type: Improvement > Components: mob >Affects Versions: 2.0.0 >Reporter: huaxiang sun >Assignee: huaxiang sun > Attachments: HBASE-17241-master-002.patch, > HBASE-17241-master-003.patch, HBASE-17241.master.001.patch > > > Today if there is only one file in the partition, and there is no _del files, > the file is skipped. With del file, the current logic is to compact the > already-compacted file with _del file. Let's say there is one mob file > regionA20161101***, which was compacted. On 12/1/2016, there is _del file > regionB20161201**_del, mob compaction kicks in, regionA20161101*** is less > than the threshold, and it is picked for compaction. Since there is a _del > file, regionA20161101 and regionB20161201***_del are compacted into > regionA20161101**_1 . After that, regionB20161201**_del cannot be deleted > since it is not a allFile compaction. The next mob compaction, > regionA20161101**_1 and regionB20161201**_del will be picked up again and be > compacted into regionA20161101***_2. So in this case, it will cause more > unnecessary IOs. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-17241) Avoid compacting already compacted mob files with _del files
[ https://issues.apache.org/jira/browse/HBASE-17241?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15719235#comment-15719235 ] huaxiang sun commented on HBASE-17241: -- Thanks Ted. Yeah, it should be getFileCount() == 1 to decrement selectedFileCount, will upload an update later. > Avoid compacting already compacted mob files with _del files > - > > Key: HBASE-17241 > URL: https://issues.apache.org/jira/browse/HBASE-17241 > Project: HBase > Issue Type: Improvement > Components: mob >Affects Versions: 2.0.0 >Reporter: huaxiang sun >Assignee: huaxiang sun > Attachments: HBASE-17241-master-002.patch, > HBASE-17241.master.001.patch > > > Today if there is only one file in the partition, and there is no _del files, > the file is skipped. With del file, the current logic is to compact the > already-compacted file with _del file. Let's say there is one mob file > regionA20161101***, which was compacted. On 12/1/2016, there is _del file > regionB20161201**_del, mob compaction kicks in, regionA20161101*** is less > than the threshold, and it is picked for compaction. Since there is a _del > file, regionA20161101 and regionB20161201***_del are compacted into > regionA20161101**_1 . After that, regionB20161201**_del cannot be deleted > since it is not a allFile compaction. The next mob compaction, > regionA20161101**_1 and regionB20161201**_del will be picked up again and be > compacted into regionA20161101***_2. So in this case, it will cause more > unnecessary IOs. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-17241) Avoid compacting already compacted mob files with _del files
[
https://issues.apache.org/jira/browse/HBASE-17241?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15718943#comment-15718943
]
Ted Yu commented on HBASE-17241:
{code}
203 if (entry.getValue().getFileCount() <= 1) {
204 it.remove();
205 --selectedFileCount;
{code}
nit: the expected getFileCount() value is 1, hence the decrement on
selectedFileCount.
> Avoid compacting already compacted mob files with _del files
> -
>
> Key: HBASE-17241
> URL: https://issues.apache.org/jira/browse/HBASE-17241
> Project: HBase
> Issue Type: Improvement
> Components: mob
>Affects Versions: 2.0.0
>Reporter: huaxiang sun
>Assignee: huaxiang sun
> Attachments: HBASE-17241-master-002.patch,
> HBASE-17241.master.001.patch
>
>
> Today if there is only one file in the partition, and there is no _del files,
> the file is skipped. With del file, the current logic is to compact the
> already-compacted file with _del file. Let's say there is one mob file
> regionA20161101***, which was compacted. On 12/1/2016, there is _del file
> regionB20161201**_del, mob compaction kicks in, regionA20161101*** is less
> than the threshold, and it is picked for compaction. Since there is a _del
> file, regionA20161101 and regionB20161201***_del are compacted into
> regionA20161101**_1 . After that, regionB20161201**_del cannot be deleted
> since it is not a allFile compaction. The next mob compaction,
> regionA20161101**_1 and regionB20161201**_del will be picked up again and be
> compacted into regionA20161101***_2. So in this case, it will cause more
> unnecessary IOs.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-17241) Avoid compacting already compacted mob files with _del files
[
https://issues.apache.org/jira/browse/HBASE-17241?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15718539#comment-15718539
]
Hadoop QA commented on HBASE-17241:
---
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 14s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
55s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 39s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
46s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
14s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
43s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 27s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
41s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 37s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 37s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
44s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
14s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
26m 10s {color} | {color:green} Patch does not cause any errors with Hadoop
2.6.1 2.6.2 2.6.3 2.6.4 2.6.5 2.7.1 2.7.2 2.7.3 or 3.0.0-alpha1. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
52s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 28s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 90m 19s
{color} | {color:green} hbase-server in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
14s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 128m 33s {color}
| {color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.12.3 Server=1.12.3 Image:yetus/hbase:8d52d23 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12841633/HBASE-17241-master-002.patch
|
| JIRA Issue | HBASE-17241 |
| Optional Tests | asflicense javac javadoc unit findbugs hadoopcheck
hbaseanti checkstyle compile |
| uname | Linux 1000a6de59a5 3.13.0-93-generic #140-Ubuntu SMP Mon Jul 18
21:21:05 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
|
| git revision | master / 03423fe |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| Test Results |
https://builds.apache.org/job/PreCommit-HBASE-Build/4775/testReport/ |
| modules | C: hbase-server U: hbase-server |
| Console output |
https://builds.apache.org/job/PreCommit-HBASE-Build/4775/console |
| Powered by | Apache Yetus 0.3.0 http://yetus.apache.org |
This message was automatically generated.
> Avoid compacting already compacted mob files with _del files
> -
>
> Key: HBASE-17241
> URL: https://issues.apache.org/jira/browse/HBASE-17241
> Project: HBase
> Issue Type: Improvement
> Components: mob
>Affects Versions: 2.0.0
>Reporter: huaxiang sun
>Assignee: huaxiang sun
> Attachments:
[jira] [Commented] (HBASE-17241) Avoid compacting already compacted mob files with _del files
[ https://issues.apache.org/jira/browse/HBASE-17241?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15716856#comment-15716856 ] huaxiang sun commented on HBASE-17241: -- Thanks Ted. The failure is caused by the patch, need to change the test case as well. Will upload a new patch. > Avoid compacting already compacted mob files with _del files > - > > Key: HBASE-17241 > URL: https://issues.apache.org/jira/browse/HBASE-17241 > Project: HBase > Issue Type: Improvement > Components: mob >Affects Versions: 2.0.0 >Reporter: huaxiang sun >Assignee: huaxiang sun > Attachments: HBASE-17241.master.001.patch > > > Today if there is only one file in the partition, and there is no _del files, > the file is skipped. With del file, the current logic is to compact the > already-compacted file with _del file. Let's say there is one mob file > regionA20161101***, which was compacted. On 12/1/2016, there is _del file > regionB20161201**_del, mob compaction kicks in, regionA20161101*** is less > than the threshold, and it is picked for compaction. Since there is a _del > file, regionA20161101 and regionB20161201***_del are compacted into > regionA20161101**_1 . After that, regionB20161201**_del cannot be deleted > since it is not a allFile compaction. The next mob compaction, > regionA20161101**_1 and regionB20161201**_del will be picked up again and be > compacted into regionA20161101***_2. So in this case, it will cause more > unnecessary IOs. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-17241) Avoid compacting already compacted mob files with _del files
[
https://issues.apache.org/jira/browse/HBASE-17241?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15716847#comment-15716847
]
Ted Yu commented on HBASE-17241:
{code}
90 public int getFileNumbers () {
{code}
Rename the method getNumberOfFiles or getFileCount
Please check the test failure of TestPartitionedMobCompactor
> Avoid compacting already compacted mob files with _del files
> -
>
> Key: HBASE-17241
> URL: https://issues.apache.org/jira/browse/HBASE-17241
> Project: HBase
> Issue Type: Improvement
> Components: mob
>Affects Versions: 2.0.0
>Reporter: huaxiang sun
>Assignee: huaxiang sun
> Attachments: HBASE-17241.master.001.patch
>
>
> Today if there is only one file in the partition, and there is no _del files,
> the file is skipped. With del file, the current logic is to compact the
> already-compacted file with _del file. Let's say there is one mob file
> regionA20161101***, which was compacted. On 12/1/2016, there is _del file
> regionB20161201**_del, mob compaction kicks in, regionA20161101*** is less
> than the threshold, and it is picked for compaction. Since there is a _del
> file, regionA20161101 and regionB20161201***_del are compacted into
> regionA20161101**_1 . After that, regionB20161201**_del cannot be deleted
> since it is not a allFile compaction. The next mob compaction,
> regionA20161101**_1 and regionB20161201**_del will be picked up again and be
> compacted into regionA20161101***_2. So in this case, it will cause more
> unnecessary IOs.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-17241) Avoid compacting already compacted mob files with _del files
[
https://issues.apache.org/jira/browse/HBASE-17241?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15715810#comment-15715810
]
Hadoop QA commented on HBASE-17241:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 18s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m 0s
{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
58s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 36s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
43s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
13s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
43s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 27s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
40s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 35s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 35s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
42s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
14s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
25m 34s {color} | {color:green} Patch does not cause any errors with Hadoop
2.6.1 2.6.2 2.6.3 2.6.4 2.6.5 2.7.1 2.7.2 2.7.3 or 3.0.0-alpha1. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
50s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 26s
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 110m 16s
{color} | {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
15s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 147m 49s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hbase.mob.compactions.TestPartitionedMobCompactor
|
| | hadoop.hbase.regionserver.TestHRegionWithInMemoryFlush |
| | hadoop.hbase.replication.TestSerialReplication |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.12.2 Server=1.12.2 Image:yetus/hbase:8d52d23 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12841513/HBASE-17241.master.001.patch
|
| JIRA Issue | HBASE-17241 |
| Optional Tests | asflicense javac javadoc unit findbugs hadoopcheck
hbaseanti checkstyle compile |
| uname | Linux eafc2e1a9d04 3.13.0-100-generic #147-Ubuntu SMP Tue Oct 18
16:48:51 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
|
| git revision | master / 9c13219 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| unit |
https://builds.apache.org/job/PreCommit-HBASE-Build/4765/artifact/patchprocess/patch-unit-hbase-server.txt
|
| unit test logs |
https://builds.apache.org/job/PreCommit-HBASE-Build/4765/artifact/patchprocess/patch-unit-hbase-server.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HBASE-Build/4765/testReport/ |
| modules | C: hbase-server U: hbase-server |
| Console output |
