[jira] [Commented] (HBASE-20064) Disable MOB threads that are running whether you MOB or not
[
https://issues.apache.org/jira/browse/HBASE-20064?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16517126#comment-16517126
]
Hadoop QA commented on HBASE-20064:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
0s{color} | {color:blue} Docker mode activated. {color} |
| {color:red}-1{color} | {color:red} patch {color} | {color:red} 0m 4s{color}
| {color:red} HBASE-20064 does not apply to master. Rebase required? Wrong
Branch? See https://yetus.apache.org/documentation/0.7.0/precommit-patchnames
for help. {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Issue | HBASE-20064 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12912215/HBASE-20064.master.002.patch
|
| Console output |
https://builds.apache.org/job/PreCommit-HBASE-Build/13311/console |
| Powered by | Apache Yetus 0.7.0 http://yetus.apache.org |
This message was automatically generated.
> Disable MOB threads that are running whether you MOB or not
> ---
>
> Key: HBASE-20064
> URL: https://issues.apache.org/jira/browse/HBASE-20064
> Project: HBase
> Issue Type: Bug
>Reporter: stack
>Assignee: Reid Chan
>Priority: Major
> Fix For: 2.2.0
>
> Attachments: HBASE-20064.master.001.patch,
> HBASE-20064.master.002.patch
>
>
> Master starts up some cleaner and compacting threads even though no MOB.
> Disable them and have users explicitly enable it.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20064) Disable MOB threads that are running whether you MOB or not
[ https://issues.apache.org/jira/browse/HBASE-20064?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16396734#comment-16396734 ] Appy commented on HBASE-20064: -- How many threads? How much work do they do when there are no MOB tables? Do you think it is worth the added code complexity and for user to enable+restart cluster to use mob? > Disable MOB threads that are running whether you MOB or not > --- > > Key: HBASE-20064 > URL: https://issues.apache.org/jira/browse/HBASE-20064 > Project: HBase > Issue Type: Bug >Reporter: stack >Assignee: Reid Chan >Priority: Major > Fix For: 2.1.0 > > Attachments: HBASE-20064.master.001.patch, > HBASE-20064.master.002.patch > > > Master starts up some cleaner and compacting threads even though no MOB. > Disable them and have users explicitly enable it. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20064) Disable MOB threads that are running whether you MOB or not
[ https://issues.apache.org/jira/browse/HBASE-20064?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16396717#comment-16396717 ] Appy commented on HBASE-20064: -- Scope is larger than what's possible between 2.0 beta2 and GA. Moving it out. Let me know if that's inaccurate... The type should be changed to Improvement (arguably) since I think [~stack] didn't push it out sooner since it was marked bug. > Disable MOB threads that are running whether you MOB or not > --- > > Key: HBASE-20064 > URL: https://issues.apache.org/jira/browse/HBASE-20064 > Project: HBase > Issue Type: Bug >Reporter: stack >Assignee: Reid Chan >Priority: Major > Fix For: 2.1.0 > > Attachments: HBASE-20064.master.001.patch, > HBASE-20064.master.002.patch > > > Master starts up some cleaner and compacting threads even though no MOB. > Disable them and have users explicitly enable it. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20064) Disable MOB threads that are running whether you MOB or not
[ https://issues.apache.org/jira/browse/HBASE-20064?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16379884#comment-16379884 ] Chia-Ping Tsai commented on HBASE-20064: Could we do the lazy instantiation for MOB thread? > Disable MOB threads that are running whether you MOB or not > --- > > Key: HBASE-20064 > URL: https://issues.apache.org/jira/browse/HBASE-20064 > Project: HBase > Issue Type: Bug >Reporter: stack >Assignee: Reid Chan >Priority: Major > Fix For: 2.0.0 > > Attachments: HBASE-20064.master.001.patch, > HBASE-20064.master.002.patch > > > Master starts up some cleaner and compacting threads even though no MOB. > Disable them and have users explicitly enable it. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20064) Disable MOB threads that are running whether you MOB or not
[
https://issues.apache.org/jira/browse/HBASE-20064?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16378344#comment-16378344
]
Hadoop QA commented on HBASE-20064:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
17s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s{color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
21s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
0s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 3m
37s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 3m
20s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 7m
31s{color} | {color:green} branch has no errors when building our shaded
downstream artifacts. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Skipped patched modules with no Java source: . {color}
|
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m
52s{color} | {color:red} hbase-server in master has 2 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 3m
8s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
12s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
54s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 3m
39s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 3m
39s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 3m
22s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
2s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 4m
15s{color} | {color:green} patch has no errors when building our shaded
downstream artifacts. {color} |
| {color:red}-1{color} | {color:red} hadoopcheck {color} | {color:red} 6m
10s{color} | {color:red} The patch causes 10 errors with Hadoop v2.6.5. {color}
|
| {color:red}-1{color} | {color:red} hadoopcheck {color} | {color:red} 8m
11s{color} | {color:red} The patch causes 10 errors with Hadoop v2.7.4. {color}
|
| {color:red}-1{color} | {color:red} hadoopcheck {color} | {color:red} 11m
5s{color} | {color:red} The patch causes 10 errors with Hadoop v3.0.0. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Skipped patched modules with no Java source: . {color}
|
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
52s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 3m
4s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green}165m
32s{color} | {color:green} root in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 1m
18s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}216m 55s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hbase:eee3b01 |
| JIRA Issue | HBASE-20064 |
| JIRA Patch URL |
[jira] [Commented] (HBASE-20064) Disable MOB threads that are running whether you MOB or not
[ https://issues.apache.org/jira/browse/HBASE-20064?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16378184#comment-16378184 ] Reid Chan commented on HBASE-20064: --- Maybe hbase need a global control component to guard the mob, this issue becomes huge... > Disable MOB threads that are running whether you MOB or not > --- > > Key: HBASE-20064 > URL: https://issues.apache.org/jira/browse/HBASE-20064 > Project: HBase > Issue Type: Bug >Reporter: stack >Assignee: Reid Chan >Priority: Major > Fix For: 2.0.0 > > Attachments: HBASE-20064.master.001.patch, > HBASE-20064.master.002.patch > > > Master starts up some cleaner and compacting threads even though no MOB. > Disable them and have users explicitly enable it. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20064) Disable MOB threads that are running whether you MOB or not
[ https://issues.apache.org/jira/browse/HBASE-20064?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16378182#comment-16378182 ] Reid Chan commented on HBASE-20064: --- Looks like there're more holes than i expected. Even though this config controls the start up, but if a user specify one column family 'MOB', then those mob files will be kept forever and have no chore to clean and have no threads to compact? > Disable MOB threads that are running whether you MOB or not > --- > > Key: HBASE-20064 > URL: https://issues.apache.org/jira/browse/HBASE-20064 > Project: HBase > Issue Type: Bug >Reporter: stack >Assignee: Reid Chan >Priority: Major > Fix For: 2.0.0 > > Attachments: HBASE-20064.master.001.patch, > HBASE-20064.master.002.patch > > > Master starts up some cleaner and compacting threads even though no MOB. > Disable them and have users explicitly enable it. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20064) Disable MOB threads that are running whether you MOB or not
[
https://issues.apache.org/jira/browse/HBASE-20064?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16378166#comment-16378166
]
Chia-Ping Tsai commented on HBASE-20064:
Use getAllDescriptors instead. The getAll will be removed by HBASE-20097.
{code:java}
+ /**
+ * Check whether any table has mob column family.
+ * @return true if there is, false otherwise
+ * @throws IOException ioe possibly happens while getting all
TableDescriptors
+ */
+ private boolean checkColumnFamilyMobEnable() throws IOException {
+TableDescriptors tds = getTableDescriptors();
+// Can we expect no ioe? since this method only called at master start up.
+for (TableDescriptor td : tds.getAll().values()) { // here
+ for (ColumnFamilyDescriptor cfd : td.getColumnFamilies()) {
+if (cfd.isMobEnabled()) {
+ return true;
+}
+ }
+}
+return false;
+ }
{code}
mobCacheConfig can be final. You can assign the null to it if the mob is
disabled.
{code:java}
- final MobCacheConfig mobCacheConfig;
+ MobCacheConfig mobCacheConfig;
{code}
Our metrics still assume the mob is enabled. Please check
RegionServerMetricsWrapperRunnable#run. It may cause NPE.
{code:java|title=RegionServerMetricsWrapperRunnable.java}
mobFileCacheAccessCount = mobFileCache.getAccessCount();
mobFileCacheMissCount = mobFileCache.getMissCount();
mobFileCacheHitRatio = Double.
isNaN(mobFileCache.getHitRatio())?0:mobFileCache.getHitRatio();
mobFileCacheEvictedCount = mobFileCache.getEvictedFileCount();
mobFileCacheCount = mobFileCache.getCacheSize();
blockedRequestsCount = tempBlockedRequestsCount;
{code}
Should we shutdown RS as with master?
{code:java}
- mobCacheConfig = new MobCacheConfig(conf);
+ if (MobUtils.isMobEnable(conf)) {
+mobCacheConfig = new MobCacheConfig(conf);
+ }
{code}
> Disable MOB threads that are running whether you MOB or not
> ---
>
> Key: HBASE-20064
> URL: https://issues.apache.org/jira/browse/HBASE-20064
> Project: HBase
> Issue Type: Bug
>Reporter: stack
>Assignee: Reid Chan
>Priority: Major
> Fix For: 2.0.0
>
> Attachments: HBASE-20064.master.001.patch,
> HBASE-20064.master.002.patch
>
>
> Master starts up some cleaner and compacting threads even though no MOB.
> Disable them and have users explicitly enable it.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20064) Disable MOB threads that are running whether you MOB or not
[ https://issues.apache.org/jira/browse/HBASE-20064?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16378125#comment-16378125 ] Reid Chan commented on HBASE-20064: --- I chose your former comment, just shutdown master and throw IllegalArgumentException. > Disable MOB threads that are running whether you MOB or not > --- > > Key: HBASE-20064 > URL: https://issues.apache.org/jira/browse/HBASE-20064 > Project: HBase > Issue Type: Bug >Reporter: stack >Assignee: Reid Chan >Priority: Major > Fix For: 2.0.0 > > Attachments: HBASE-20064.master.001.patch, > HBASE-20064.master.002.patch > > > Master starts up some cleaner and compacting threads even though no MOB. > Disable them and have users explicitly enable it. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20064) Disable MOB threads that are running whether you MOB or not
[ https://issues.apache.org/jira/browse/HBASE-20064?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16376895#comment-16376895 ] Reid Chan commented on HBASE-20064: --- Got your point, let me think... > Disable MOB threads that are running whether you MOB or not > --- > > Key: HBASE-20064 > URL: https://issues.apache.org/jira/browse/HBASE-20064 > Project: HBase > Issue Type: Bug >Reporter: stack >Assignee: Reid Chan >Priority: Major > Fix For: 2.0.0 > > Attachments: HBASE-20064.master.001.patch > > > Master starts up some cleaner and compacting threads even though no MOB. > Disable them and have users explicitly enable it. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20064) Disable MOB threads that are running whether you MOB or not
[ https://issues.apache.org/jira/browse/HBASE-20064?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16376880#comment-16376880 ] Chia-Ping Tsai commented on HBASE-20064: Should we shutdown the master if there is a mob table already? Or just ignore the new config and log debug message? > Disable MOB threads that are running whether you MOB or not > --- > > Key: HBASE-20064 > URL: https://issues.apache.org/jira/browse/HBASE-20064 > Project: HBase > Issue Type: Bug >Reporter: stack >Assignee: Reid Chan >Priority: Major > Fix For: 2.0.0 > > Attachments: HBASE-20064.master.001.patch > > > Master starts up some cleaner and compacting threads even though no MOB. > Disable them and have users explicitly enable it. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20064) Disable MOB threads that are running whether you MOB or not
[
https://issues.apache.org/jira/browse/HBASE-20064?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16376773#comment-16376773
]
Hadoop QA commented on HBASE-20064:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
10s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s{color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
34s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
42s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 4m
12s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 3m
48s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 9m
19s{color} | {color:green} branch has no errors when building our shaded
downstream artifacts. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Skipped patched modules with no Java source: . {color}
|
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m
6s{color} | {color:red} hbase-common in master has 2 extant Findbugs warnings.
{color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 3m
15s{color} | {color:red} hbase-server in master has 22 extant Findbugs
warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 5m
54s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
14s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
24s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 4m
43s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 4m
43s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 3m
54s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
2s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 4m
43s{color} | {color:green} patch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
18m 21s{color} | {color:green} Patch does not cause any errors with Hadoop
2.6.5 2.7.4 or 3.0.0. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Skipped patched modules with no Java source: . {color}
|
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
44s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 3m
14s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red}132m 38s{color}
| {color:red} root in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 1m
40s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}204m 15s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hbase.mob.compactions.TestMobCompactor |
| | hadoop.hbase.client.TestAsyncRegionAdminApi |
\\
\\
|| Subsystem || Report/Notes
