[jira] [Commented] (HBASE-14969) Add throughput controller for flush
[
https://issues.apache.org/jira/browse/HBASE-14969?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15202885#comment-15202885
]
Yu Li commented on HBASE-14969:
---
Thank you for the double check sir [~ghelmling]
bq. If the change wasn't intentional, maybe we should do a small follow up
issue to change the default back. Even better if we remove the duplicate class
and just use one.
Yes you are right, it was my fault not being carefully enough when preparing
the patch for branch-1 (sorry about this...). The one in o.a.h.h.compactions
should be removed, and we should change the default back to
{{NoLimitThroughputController}} for branch-1. I'll open another JIRA to fix
this, and thank you for pointing out the issue.
> Add throughput controller for flush
> ---
>
> Key: HBASE-14969
> URL: https://issues.apache.org/jira/browse/HBASE-14969
> Project: HBase
> Issue Type: Improvement
> Components: regionserver
>Affects Versions: 2.0.0, 1.3.0
>Reporter: Yu Li
>Assignee: Yu Li
> Fix For: 2.0.0, 1.3.0
>
> Attachments: HBASE-14969-addendum-branch-1.patch,
> HBASE-14969.branch-1.patch, HBASE-14969.patch, HBASE-14969_v10.patch,
> HBASE-14969_v2.patch, HBASE-14969_v3.patch, HBASE-14969_v4.patch,
> HBASE-14969_v5.patch, HBASE-14969_v6.patch, HBASE-14969_v9.patch,
> fd78628_9909808_compat_report.html, load-nothrottling.log, load-throttling.log
>
>
> In HBASE-8329 we added a throughput controller for compaction, to avoid spike
> caused by huge IO pressure like network/disk overflow. However, even with
> this control on, we are still observing disk utils near 100%, and by analysis
> we think this is caused by flush, especially when we increase the setting of
> {{hbase.hstore.flusher.count}}
> In this JIRA, we propose to add throughput control feature for flush, as a
> supplement of HBASE-8329 to better control IO pressure.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14969) Add throughput controller for flush
[
https://issues.apache.org/jira/browse/HBASE-14969?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15202229#comment-15202229
]
Gary Helmling commented on HBASE-14969:
---
This change switches the default compaction throughput controller to
PressureAwareCompactionThroughputController in 1.3, which limits throughput to
an upper bound of 20MB by default, which is a pretty big change operationally
for anyone who was previously running with the default of unlimited compactions.
I think we need a release node identifying this change to the default
compaction throughput controller as part of this issue, and we may want to also
re-evaluate if the default lower and upper bound for
PressureAwareCompactionThroughputController are too low.
> Add throughput controller for flush
> ---
>
> Key: HBASE-14969
> URL: https://issues.apache.org/jira/browse/HBASE-14969
> Project: HBase
> Issue Type: Improvement
> Components: regionserver
>Affects Versions: 2.0.0, 1.3.0
>Reporter: Yu Li
>Assignee: Yu Li
> Fix For: 2.0.0, 1.3.0
>
> Attachments: HBASE-14969-addendum-branch-1.patch,
> HBASE-14969.branch-1.patch, HBASE-14969.patch, HBASE-14969_v10.patch,
> HBASE-14969_v2.patch, HBASE-14969_v3.patch, HBASE-14969_v4.patch,
> HBASE-14969_v5.patch, HBASE-14969_v6.patch, HBASE-14969_v9.patch,
> fd78628_9909808_compat_report.html, load-nothrottling.log, load-throttling.log
>
>
> In HBASE-8329 we added a throughput controller for compaction, to avoid spike
> caused by huge IO pressure like network/disk overflow. However, even with
> this control on, we are still observing disk utils near 100%, and by analysis
> we think this is caused by flush, especially when we increase the setting of
> {{hbase.hstore.flusher.count}}
> In this JIRA, we propose to add throughput control feature for flush, as a
> supplement of HBASE-8329 to better control IO pressure.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14969) Add throughput controller for flush
[
https://issues.apache.org/jira/browse/HBASE-14969?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15202854#comment-15202854
]
Gary Helmling commented on HBASE-14969:
---
[~carp84], the part I'm looking at is in {{CompactSplitThread}}, where the
{{CompactionThroughputControllerFactory}} used changed:
{code}
-import
org.apache.hadoop.hbase.regionserver.compactions.CompactionThroughputController;
-import
org.apache.hadoop.hbase.regionserver.compactions.CompactionThroughputControllerFactory;
+import
org.apache.hadoop.hbase.regionserver.throttle.CompactionThroughputControllerFactory;
+import org.apache.hadoop.hbase.regionserver.throttle.ThroughputController;
{code}
The snippet you pulled is from
o.a.h.h.regionserver.compactions.CompactionThroughputController.
I don't know why we have two {{CompactionThroughputControllerFactory}} classes,
but the o.a.h.h.regionserver.throttle one defaults to
{{PressureAwareCompactionThroughputController}} while the
o.a.h.h.regionserver.compactions one (previously used) defaults to
{{NoLimitCompactionThroughputController}}.
If the change wasn't intentional, maybe we should do a small follow up issue to
change the default back. Even better if we remove the duplicate class and just
use one.
> Add throughput controller for flush
> ---
>
> Key: HBASE-14969
> URL: https://issues.apache.org/jira/browse/HBASE-14969
> Project: HBase
> Issue Type: Improvement
> Components: regionserver
>Affects Versions: 2.0.0, 1.3.0
>Reporter: Yu Li
>Assignee: Yu Li
> Fix For: 2.0.0, 1.3.0
>
> Attachments: HBASE-14969-addendum-branch-1.patch,
> HBASE-14969.branch-1.patch, HBASE-14969.patch, HBASE-14969_v10.patch,
> HBASE-14969_v2.patch, HBASE-14969_v3.patch, HBASE-14969_v4.patch,
> HBASE-14969_v5.patch, HBASE-14969_v6.patch, HBASE-14969_v9.patch,
> fd78628_9909808_compat_report.html, load-nothrottling.log, load-throttling.log
>
>
> In HBASE-8329 we added a throughput controller for compaction, to avoid spike
> caused by huge IO pressure like network/disk overflow. However, even with
> this control on, we are still observing disk utils near 100%, and by analysis
> we think this is caused by flush, especially when we increase the setting of
> {{hbase.hstore.flusher.count}}
> In this JIRA, we propose to add throughput control feature for flush, as a
> supplement of HBASE-8329 to better control IO pressure.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14969) Add throughput controller for flush
[
https://issues.apache.org/jira/browse/HBASE-14969?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15202707#comment-15202707
]
Yu Li commented on HBASE-14969:
---
bq. This change switches the default compaction throughput controller to
PressureAwareCompactionThroughputController in 1.3, which limits throughput to
an upper bound of 20MB by default
Checking the commit and it seemed to me the default compaction throughput
controller is still the no limit one, excerpt from
{{CompactionThroughputControllerFactory}}:
{code}
private static final Class
DEFAULT_THROUGHPUT_CONTROLLER_CLASS = NoLimitThroughputController.class;
{code}
Mind give it a double check sir [~ghelmling]? Thanks.
On the other hand, since this is kind of a new feature and did some refactor,
yes let me prepare one release note.
> Add throughput controller for flush
> ---
>
> Key: HBASE-14969
> URL: https://issues.apache.org/jira/browse/HBASE-14969
> Project: HBase
> Issue Type: Improvement
> Components: regionserver
>Affects Versions: 2.0.0, 1.3.0
>Reporter: Yu Li
>Assignee: Yu Li
> Fix For: 2.0.0, 1.3.0
>
> Attachments: HBASE-14969-addendum-branch-1.patch,
> HBASE-14969.branch-1.patch, HBASE-14969.patch, HBASE-14969_v10.patch,
> HBASE-14969_v2.patch, HBASE-14969_v3.patch, HBASE-14969_v4.patch,
> HBASE-14969_v5.patch, HBASE-14969_v6.patch, HBASE-14969_v9.patch,
> fd78628_9909808_compat_report.html, load-nothrottling.log, load-throttling.log
>
>
> In HBASE-8329 we added a throughput controller for compaction, to avoid spike
> caused by huge IO pressure like network/disk overflow. However, even with
> this control on, we are still observing disk utils near 100%, and by analysis
> we think this is caused by flush, especially when we increase the setting of
> {{hbase.hstore.flusher.count}}
> In this JIRA, we propose to add throughput control feature for flush, as a
> supplement of HBASE-8329 to better control IO pressure.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14969) Add throughput controller for flush
[
https://issues.apache.org/jira/browse/HBASE-14969?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15126854#comment-15126854
]
Hudson commented on HBASE-14969:
FAILURE: Integrated in HBase-Trunk_matrix #673 (See
[https://builds.apache.org/job/HBase-Trunk_matrix/673/])
HBASE-14969 Add throughput controller for flush; ADDENDUM (stack: rev
fc5e698c0a4e9613ab9eddcfc2a2684424806c56)
*
hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/throttle/TestFlushWithThroughputController.java
> Add throughput controller for flush
> ---
>
> Key: HBASE-14969
> URL: https://issues.apache.org/jira/browse/HBASE-14969
> Project: HBase
> Issue Type: Improvement
> Components: regionserver
>Affects Versions: 2.0.0, 1.3.0
>Reporter: Yu Li
>Assignee: Yu Li
> Fix For: 2.0.0, 1.3.0
>
> Attachments: HBASE-14969-addendum-branch-1.patch,
> HBASE-14969.branch-1.patch, HBASE-14969.patch, HBASE-14969_v10.patch,
> HBASE-14969_v2.patch, HBASE-14969_v3.patch, HBASE-14969_v4.patch,
> HBASE-14969_v5.patch, HBASE-14969_v6.patch, HBASE-14969_v9.patch,
> fd78628_9909808_compat_report.html, load-nothrottling.log, load-throttling.log
>
>
> In HBASE-8329 we added a throughput controller for compaction, to avoid spike
> caused by huge IO pressure like network/disk overflow. However, even with
> this control on, we are still observing disk utils near 100%, and by analysis
> we think this is caused by flush, especially when we increase the setting of
> {{hbase.hstore.flusher.count}}
> In this JIRA, we propose to add throughput control feature for flush, as a
> supplement of HBASE-8329 to better control IO pressure.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14969) Add throughput controller for flush
[
https://issues.apache.org/jira/browse/HBASE-14969?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15126829#comment-15126829
]
Hudson commented on HBASE-14969:
SUCCESS: Integrated in HBase-1.3-IT #469 (See
[https://builds.apache.org/job/HBase-1.3-IT/469/])
HBASE-14969 Add throughput controller for flush; ADDENDUM (stack: rev
3cc80e37c6c4e3c94b531340f96716c29d02bd12)
*
hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/throttle/TestFlushWithThroughputController.java
> Add throughput controller for flush
> ---
>
> Key: HBASE-14969
> URL: https://issues.apache.org/jira/browse/HBASE-14969
> Project: HBase
> Issue Type: Improvement
> Components: regionserver
>Affects Versions: 2.0.0, 1.3.0
>Reporter: Yu Li
>Assignee: Yu Li
> Fix For: 2.0.0, 1.3.0
>
> Attachments: HBASE-14969-addendum-branch-1.patch,
> HBASE-14969.branch-1.patch, HBASE-14969.patch, HBASE-14969_v10.patch,
> HBASE-14969_v2.patch, HBASE-14969_v3.patch, HBASE-14969_v4.patch,
> HBASE-14969_v5.patch, HBASE-14969_v6.patch, HBASE-14969_v9.patch,
> fd78628_9909808_compat_report.html, load-nothrottling.log, load-throttling.log
>
>
> In HBASE-8329 we added a throughput controller for compaction, to avoid spike
> caused by huge IO pressure like network/disk overflow. However, even with
> this control on, we are still observing disk utils near 100%, and by analysis
> we think this is caused by flush, especially when we increase the setting of
> {{hbase.hstore.flusher.count}}
> In this JIRA, we propose to add throughput control feature for flush, as a
> supplement of HBASE-8329 to better control IO pressure.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14969) Add throughput controller for flush
[
https://issues.apache.org/jira/browse/HBASE-14969?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15127517#comment-15127517
]
Yu Li commented on HBASE-14969:
---
bq. so where is the actual failure that we can see?
[~ram_krish] I saw the TestBlockEvictionFromClient failure from the [jdk8 UT
result|https://builds.apache.org/job/PreCommit-HBASE-Build/367/artifact/patchprocess/patch-unit-hbase-server-jdk1.8.0_66.txt],
FYI.
> Add throughput controller for flush
> ---
>
> Key: HBASE-14969
> URL: https://issues.apache.org/jira/browse/HBASE-14969
> Project: HBase
> Issue Type: Improvement
> Components: regionserver
>Affects Versions: 2.0.0, 1.3.0
>Reporter: Yu Li
>Assignee: Yu Li
> Fix For: 2.0.0, 1.3.0
>
> Attachments: HBASE-14969-addendum-branch-1.patch,
> HBASE-14969.branch-1.patch, HBASE-14969.patch, HBASE-14969_v10.patch,
> HBASE-14969_v2.patch, HBASE-14969_v3.patch, HBASE-14969_v4.patch,
> HBASE-14969_v5.patch, HBASE-14969_v6.patch, HBASE-14969_v9.patch,
> fd78628_9909808_compat_report.html, load-nothrottling.log, load-throttling.log
>
>
> In HBASE-8329 we added a throughput controller for compaction, to avoid spike
> caused by huge IO pressure like network/disk overflow. However, even with
> this control on, we are still observing disk utils near 100%, and by analysis
> we think this is caused by flush, especially when we increase the setting of
> {{hbase.hstore.flusher.count}}
> In this JIRA, we propose to add throughput control feature for flush, as a
> supplement of HBASE-8329 to better control IO pressure.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14969) Add throughput controller for flush
[
https://issues.apache.org/jira/browse/HBASE-14969?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15125915#comment-15125915
]
Duo Zhang commented on HBASE-14969:
---
[~stack] Yes, {{testFlushThroughputTuning}} fails consistently under java8. Let
me finger out why.
Thanks.
> Add throughput controller for flush
> ---
>
> Key: HBASE-14969
> URL: https://issues.apache.org/jira/browse/HBASE-14969
> Project: HBase
> Issue Type: Improvement
> Components: regionserver
>Affects Versions: 2.0.0, 1.3.0
>Reporter: Yu Li
>Assignee: Yu Li
> Fix For: 2.0.0, 1.3.0
>
> Attachments: HBASE-14969.branch-1.patch, HBASE-14969.patch,
> HBASE-14969_v10.patch, HBASE-14969_v2.patch, HBASE-14969_v3.patch,
> HBASE-14969_v4.patch, HBASE-14969_v5.patch, HBASE-14969_v6.patch,
> HBASE-14969_v9.patch, fd78628_9909808_compat_report.html,
> load-nothrottling.log, load-throttling.log
>
>
> In HBASE-8329 we added a throughput controller for compaction, to avoid spike
> caused by huge IO pressure like network/disk overflow. However, even with
> this control on, we are still observing disk utils near 100%, and by analysis
> we think this is caused by flush, especially when we increase the setting of
> {{hbase.hstore.flusher.count}}
> In this JIRA, we propose to add throughput control feature for flush, as a
> supplement of HBASE-8329 to better control IO pressure.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14969) Add throughput controller for flush
[
https://issues.apache.org/jira/browse/HBASE-14969?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15125931#comment-15125931
]
Duo Zhang commented on HBASE-14969:
---
The problem here is that {{hbase:meta}} is assigned to the RS so its
{{flushPressure}} is not 0 and cause the throughput limit differ from lower
bound.
It is only a testcase issue, can be fixed with an addendum. Let me try.
Thanks.
> Add throughput controller for flush
> ---
>
> Key: HBASE-14969
> URL: https://issues.apache.org/jira/browse/HBASE-14969
> Project: HBase
> Issue Type: Improvement
> Components: regionserver
>Affects Versions: 2.0.0, 1.3.0
>Reporter: Yu Li
>Assignee: Yu Li
> Fix For: 2.0.0, 1.3.0
>
> Attachments: HBASE-14969.branch-1.patch, HBASE-14969.patch,
> HBASE-14969_v10.patch, HBASE-14969_v2.patch, HBASE-14969_v3.patch,
> HBASE-14969_v4.patch, HBASE-14969_v5.patch, HBASE-14969_v6.patch,
> HBASE-14969_v9.patch, fd78628_9909808_compat_report.html,
> load-nothrottling.log, load-throttling.log
>
>
> In HBASE-8329 we added a throughput controller for compaction, to avoid spike
> caused by huge IO pressure like network/disk overflow. However, even with
> this control on, we are still observing disk utils near 100%, and by analysis
> we think this is caused by flush, especially when we increase the setting of
> {{hbase.hstore.flusher.count}}
> In this JIRA, we propose to add throughput control feature for flush, as a
> supplement of HBASE-8329 to better control IO pressure.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14969) Add throughput controller for flush
[
https://issues.apache.org/jira/browse/HBASE-14969?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15125993#comment-15125993
]
Yu Li commented on HBASE-14969:
---
It was my fault not making the test robust enough, sorry sir [~stack].
Thanks for the quick action [~Apache9]. I think we also need to include the
{{assertEquals}} check in {{testFlushThroughputTuning}} into the try clause to
make sure the minicluster will be shutdown at end of the case:
{code}
assertEquals(10L * 1024 * 1024,
((PressureAwareThroughputController)
TEST_UTIL.getMiniHBaseCluster().getRegionServer(0)
.getFlushThroughputController()).getMaxThroughput(), EPSILON);
Connection conn = ConnectionFactory.createConnection(conf);
try {
HTableDescriptor htd = new HTableDescriptor(tableName);
htd.addFamily(new HColumnDescriptor(family));
{code}
Let me prepare an addendum for you to review.
> Add throughput controller for flush
> ---
>
> Key: HBASE-14969
> URL: https://issues.apache.org/jira/browse/HBASE-14969
> Project: HBase
> Issue Type: Improvement
> Components: regionserver
>Affects Versions: 2.0.0, 1.3.0
>Reporter: Yu Li
>Assignee: Yu Li
> Fix For: 2.0.0, 1.3.0
>
> Attachments: HBASE-14969.branch-1.patch, HBASE-14969.patch,
> HBASE-14969_v10.patch, HBASE-14969_v2.patch, HBASE-14969_v3.patch,
> HBASE-14969_v4.patch, HBASE-14969_v5.patch, HBASE-14969_v6.patch,
> HBASE-14969_v9.patch, fd78628_9909808_compat_report.html,
> load-nothrottling.log, load-throttling.log
>
>
> In HBASE-8329 we added a throughput controller for compaction, to avoid spike
> caused by huge IO pressure like network/disk overflow. However, even with
> this control on, we are still observing disk utils near 100%, and by analysis
> we think this is caused by flush, especially when we increase the setting of
> {{hbase.hstore.flusher.count}}
> In this JIRA, we propose to add throughput control feature for flush, as a
> supplement of HBASE-8329 to better control IO pressure.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14969) Add throughput controller for flush
[
https://issues.apache.org/jira/browse/HBASE-14969?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15126042#comment-15126042
]
Yu Li commented on HBASE-14969:
---
The addendum patch lgtm, thanks Duo!
> Add throughput controller for flush
> ---
>
> Key: HBASE-14969
> URL: https://issues.apache.org/jira/browse/HBASE-14969
> Project: HBase
> Issue Type: Improvement
> Components: regionserver
>Affects Versions: 2.0.0, 1.3.0
>Reporter: Yu Li
>Assignee: Yu Li
> Fix For: 2.0.0, 1.3.0
>
> Attachments: HBASE-14969-addendum-branch-1.patch,
> HBASE-14969.branch-1.patch, HBASE-14969.patch, HBASE-14969_v10.patch,
> HBASE-14969_v2.patch, HBASE-14969_v3.patch, HBASE-14969_v4.patch,
> HBASE-14969_v5.patch, HBASE-14969_v6.patch, HBASE-14969_v9.patch,
> fd78628_9909808_compat_report.html, load-nothrottling.log, load-throttling.log
>
>
> In HBASE-8329 we added a throughput controller for compaction, to avoid spike
> caused by huge IO pressure like network/disk overflow. However, even with
> this control on, we are still observing disk utils near 100%, and by analysis
> we think this is caused by flush, especially when we increase the setting of
> {{hbase.hstore.flusher.count}}
> In this JIRA, we propose to add throughput control feature for flush, as a
> supplement of HBASE-8329 to better control IO pressure.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14969) Add throughput controller for flush
[
https://issues.apache.org/jira/browse/HBASE-14969?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15126273#comment-15126273
]
Hadoop QA commented on HBASE-14969:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 0s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
52s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 42s
{color} | {color:green} master passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 40s
{color} | {color:green} master passed with JDK v1.7.0_91 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 4m
35s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
18s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
12s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 31s
{color} | {color:green} master passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 41s
{color} | {color:green} master passed with JDK v1.7.0_91 {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
52s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 41s
{color} | {color:green} the patch passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 41s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 39s
{color} | {color:green} the patch passed with JDK v1.7.0_91 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 39s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 4m
43s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
20s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
26m 11s {color} | {color:green} Patch does not cause any errors with Hadoop
2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.1 2.6.2 2.6.3 2.7.1. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
40s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 35s
{color} | {color:green} the patch passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 40s
{color} | {color:green} the patch passed with JDK v1.7.0_91 {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 104m 59s
{color} | {color:red} hbase-server in the patch failed with JDK v1.8.0_66.
{color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 99m 42s {color}
| {color:red} hbase-server in the patch failed with JDK v1.7.0_91. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
16s {color} | {color:green} Patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 255m 23s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| JDK v1.8.0_66 Failed junit tests |
hadoop.hbase.client.TestBlockEvictionFromClient |
| JDK v1.8.0_66 Timed out junit tests |
org.apache.hadoop.hbase.client.TestHTableMultiplexerFlushCache |
| JDK v1.7.0_91 Failed junit tests |
hadoop.hbase.regionserver.TestRegionMergeTransactionOnCluster |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.9.1 Server=1.9.1 Image:yetus/hbase:date2016-02-01 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12785481/HBASE-14969-addendum-branch-1.patch
|
| JIRA Issue |
[jira] [Commented] (HBASE-14969) Add throughput controller for flush
[
https://issues.apache.org/jira/browse/HBASE-14969?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15126518#comment-15126518
]
ramkrishna.s.vasudevan commented on HBASE-14969:
In this report -
https://builds.apache.org/job/PreCommit-HBASE-Build/367/testReport/org.apache.hadoop.hbase.client/TestBlockEvictionFromClient/
seems to have passed.
So where is the actual failure that we can see?
> Add throughput controller for flush
> ---
>
> Key: HBASE-14969
> URL: https://issues.apache.org/jira/browse/HBASE-14969
> Project: HBase
> Issue Type: Improvement
> Components: regionserver
>Affects Versions: 2.0.0, 1.3.0
>Reporter: Yu Li
>Assignee: Yu Li
> Fix For: 2.0.0, 1.3.0
>
> Attachments: HBASE-14969-addendum-branch-1.patch,
> HBASE-14969.branch-1.patch, HBASE-14969.patch, HBASE-14969_v10.patch,
> HBASE-14969_v2.patch, HBASE-14969_v3.patch, HBASE-14969_v4.patch,
> HBASE-14969_v5.patch, HBASE-14969_v6.patch, HBASE-14969_v9.patch,
> fd78628_9909808_compat_report.html, load-nothrottling.log, load-throttling.log
>
>
> In HBASE-8329 we added a throughput controller for compaction, to avoid spike
> caused by huge IO pressure like network/disk overflow. However, even with
> this control on, we are still observing disk utils near 100%, and by analysis
> we think this is caused by flush, especially when we increase the setting of
> {{hbase.hstore.flusher.count}}
> In this JIRA, we propose to add throughput control feature for flush, as a
> supplement of HBASE-8329 to better control IO pressure.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14969) Add throughput controller for flush
[
https://issues.apache.org/jira/browse/HBASE-14969?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15127622#comment-15127622
]
ramkrishna.s.vasudevan commented on HBASE-14969:
[~carp84]
Thanks for the info. Let me see the reason for the failure but the same passes
in the other test report. There should be some flakiness.
> Add throughput controller for flush
> ---
>
> Key: HBASE-14969
> URL: https://issues.apache.org/jira/browse/HBASE-14969
> Project: HBase
> Issue Type: Improvement
> Components: regionserver
>Affects Versions: 2.0.0, 1.3.0
>Reporter: Yu Li
>Assignee: Yu Li
> Fix For: 2.0.0, 1.3.0
>
> Attachments: HBASE-14969-addendum-branch-1.patch,
> HBASE-14969.branch-1.patch, HBASE-14969.patch, HBASE-14969_v10.patch,
> HBASE-14969_v2.patch, HBASE-14969_v3.patch, HBASE-14969_v4.patch,
> HBASE-14969_v5.patch, HBASE-14969_v6.patch, HBASE-14969_v9.patch,
> fd78628_9909808_compat_report.html, load-nothrottling.log, load-throttling.log
>
>
> In HBASE-8329 we added a throughput controller for compaction, to avoid spike
> caused by huge IO pressure like network/disk overflow. However, even with
> this control on, we are still observing disk utils near 100%, and by analysis
> we think this is caused by flush, especially when we increase the setting of
> {{hbase.hstore.flusher.count}}
> In this JIRA, we propose to add throughput control feature for flush, as a
> supplement of HBASE-8329 to better control IO pressure.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14969) Add throughput controller for flush
[
https://issues.apache.org/jira/browse/HBASE-14969?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15126746#comment-15126746
]
Hudson commented on HBASE-14969:
FAILURE: Integrated in HBase-1.3 #526 (See
[https://builds.apache.org/job/HBase-1.3/526/])
HBASE-14969 Add throughput controller for flush; ADDENDUM (stack: rev
3cc80e37c6c4e3c94b531340f96716c29d02bd12)
*
hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/throttle/TestFlushWithThroughputController.java
> Add throughput controller for flush
> ---
>
> Key: HBASE-14969
> URL: https://issues.apache.org/jira/browse/HBASE-14969
> Project: HBase
> Issue Type: Improvement
> Components: regionserver
>Affects Versions: 2.0.0, 1.3.0
>Reporter: Yu Li
>Assignee: Yu Li
> Fix For: 2.0.0, 1.3.0
>
> Attachments: HBASE-14969-addendum-branch-1.patch,
> HBASE-14969.branch-1.patch, HBASE-14969.patch, HBASE-14969_v10.patch,
> HBASE-14969_v2.patch, HBASE-14969_v3.patch, HBASE-14969_v4.patch,
> HBASE-14969_v5.patch, HBASE-14969_v6.patch, HBASE-14969_v9.patch,
> fd78628_9909808_compat_report.html, load-nothrottling.log, load-throttling.log
>
>
> In HBASE-8329 we added a throughput controller for compaction, to avoid spike
> caused by huge IO pressure like network/disk overflow. However, even with
> this control on, we are still observing disk utils near 100%, and by analysis
> we think this is caused by flush, especially when we increase the setting of
> {{hbase.hstore.flusher.count}}
> In this JIRA, we propose to add throughput control feature for flush, as a
> supplement of HBASE-8329 to better control IO pressure.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14969) Add throughput controller for flush
[
https://issues.apache.org/jira/browse/HBASE-14969?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15126301#comment-15126301
]
Yu Li commented on HBASE-14969:
---
Checking the UT result, everything looks ok except for those known flaky
failures like {{TestRegionMergeTransactionOnCluster.testCleanMergeReference}}
and {{TestBlockEvictionFromClient.testMultiGets}}
> Add throughput controller for flush
> ---
>
> Key: HBASE-14969
> URL: https://issues.apache.org/jira/browse/HBASE-14969
> Project: HBase
> Issue Type: Improvement
> Components: regionserver
>Affects Versions: 2.0.0, 1.3.0
>Reporter: Yu Li
>Assignee: Yu Li
> Fix For: 2.0.0, 1.3.0
>
> Attachments: HBASE-14969-addendum-branch-1.patch,
> HBASE-14969.branch-1.patch, HBASE-14969.patch, HBASE-14969_v10.patch,
> HBASE-14969_v2.patch, HBASE-14969_v3.patch, HBASE-14969_v4.patch,
> HBASE-14969_v5.patch, HBASE-14969_v6.patch, HBASE-14969_v9.patch,
> fd78628_9909808_compat_report.html, load-nothrottling.log, load-throttling.log
>
>
> In HBASE-8329 we added a throughput controller for compaction, to avoid spike
> caused by huge IO pressure like network/disk overflow. However, even with
> this control on, we are still observing disk utils near 100%, and by analysis
> we think this is caused by flush, especially when we increase the setting of
> {{hbase.hstore.flusher.count}}
> In this JIRA, we propose to add throughput control feature for flush, as a
> supplement of HBASE-8329 to better control IO pressure.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14969) Add throughput controller for flush
[ https://issues.apache.org/jira/browse/HBASE-14969?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15123148#comment-15123148 ] Hudson commented on HBASE-14969: SUCCESS: Integrated in HBase-Trunk_matrix #667 (See [https://builds.apache.org/job/HBase-Trunk_matrix/667/]) HBASE-14969 Add throughput controller for flush (zhangduo: rev b3b1ce99c63d79401ddda9c114850dea61af0afb) * hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/throttle/PressureAwareFlushThroughputController.java * hbase-server/src/main/java/org/apache/hadoop/hbase/mob/DefaultMobStoreFlusher.java * hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestStripeCompactor.java * hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java * hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestSplitTransactionOnCluster.java * hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/StripeStoreEngine.java * hbase-server/src/test/java/org/apache/hadoop/hbase/master/MockRegionServer.java * hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/throttle/FlushThroughputControllerFactory.java * hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/compactions/Compactor.java * hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java * hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestStore.java * hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestHRegionReplayEvents.java * hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/compactions/StripeCompactionPolicy.java * hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/compactions/CompactionThroughputControllerFactory.java * hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestWALReplay.java * hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestCompactSplitThread.java * hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/compactions/NoLimitCompactionThroughputController.java * hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/StoreFlusher.java * hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/throttle/TestFlushWithThroughputController.java * hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/compactions/TestCompactionWithThroughputController.java * hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/throttle/PressureAwareThroughputController.java * hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/DefaultStoreEngine.java * hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/compactions/TestStripeCompactionPolicy.java * hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/Store.java * hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/StripeStoreFlusher.java * hbase-server/src/test/java/org/apache/hadoop/hbase/TestIOFencing.java * hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/CompactionTool.java * hbase-server/src/test/java/org/apache/hadoop/hbase/MockRegionServerServices.java * hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/compactions/CompactionThroughputController.java * hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestHMobStore.java * hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestStripeStoreEngine.java * hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/compactions/CompactionContext.java * hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/throttle/PressureAwareCompactionThroughputController.java * hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/DefaultStoreFlusher.java * hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/compactions/StripeCompactor.java * hbase-server/src/test/java/org/apache/hadoop/hbase/coprocessor/TestRegionObserverScannerOpenHook.java * hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HStore.java * hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/throttle/TestCompactionWithThroughputController.java * hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/compactions/PressureAwareCompactionThroughputController.java * hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/throttle/ThroughputControlUtil.java * hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/throttle/ThroughputController.java * hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/throttle/CompactionThroughputControllerFactory.java * hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/RegionServerServices.java * hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HMobStore.java * hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestCompaction.java * hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/CompactSplitThread.java *
[jira] [Commented] (HBASE-14969) Add throughput controller for flush
[ https://issues.apache.org/jira/browse/HBASE-14969?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15123068#comment-15123068 ] Hudson commented on HBASE-14969: FAILURE: Integrated in HBase-1.3 #522 (See [https://builds.apache.org/job/HBase-1.3/522/]) HBASE-14969 Add throughput controller for flush (zhangduo: rev 0d21fa92791ae7d704f48311539facba7061770b) * hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/compactions/StripeCompactor.java * hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestWALReplay.java * hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/throttle/PressureAwareCompactionThroughputController.java * hbase-server/src/test/java/org/apache/hadoop/hbase/master/MockRegionServer.java * hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/throttle/PressureAwareFlushThroughputController.java * hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/RegionServerServices.java * hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HStore.java * hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/DefaultStoreFlusher.java * hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/compactions/StripeCompactionPolicy.java * hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/compactions/CompactionThroughputControllerFactory.java * hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/throttle/CompactionThroughputControllerFactory.java * hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/compactions/TestStripeCompactionPolicy.java * hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/compactions/CompactionThroughputController.java * hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/compactions/PressureAwareCompactionThroughputController.java * hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/throttle/ThroughputControlUtil.java * hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java * hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/compactions/NoLimitCompactionThroughputController.java * hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/StripeStoreEngine.java * hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/throttle/FlushThroughputControllerFactory.java * hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/StripeStoreFlusher.java * hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/throttle/NoLimitThroughputController.java * hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/throttle/TestCompactionWithThroughputController.java * hbase-server/src/test/java/org/apache/hadoop/hbase/coprocessor/TestRegionObserverScannerOpenHook.java * hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/compactions/DefaultCompactor.java * hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/compactions/TestCompactionWithThroughputController.java * hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/compactions/CompactionContext.java * hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestCompactSplitThread.java * hbase-server/src/test/java/org/apache/hadoop/hbase/TestIOFencing.java * hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/CompactionTool.java * hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/throttle/PressureAwareThroughputController.java * hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestSplitTransactionOnCluster.java * hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestCompaction.java * hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/StoreFlusher.java * hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/CompactSplitThread.java * hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/DefaultStoreEngine.java * hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/Store.java * hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/throttle/ThroughputController.java * hbase-server/src/test/java/org/apache/hadoop/hbase/MockRegionServerServices.java * hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/throttle/TestFlushWithThroughputController.java * hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestStripeCompactor.java * hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java * hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestStore.java * hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestHRegionReplayEvents.java * hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestStripeStoreEngine.java * hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/compactions/Compactor.java > Add throughput controller for flush > --- > >
[jira] [Commented] (HBASE-14969) Add throughput controller for flush
[
https://issues.apache.org/jira/browse/HBASE-14969?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15122759#comment-15122759
]
Yu Li commented on HBASE-14969:
---
Thanks [~tedyu] for review and thanks [~Apache9] for review and help commit.
> Add throughput controller for flush
> ---
>
> Key: HBASE-14969
> URL: https://issues.apache.org/jira/browse/HBASE-14969
> Project: HBase
> Issue Type: Improvement
> Components: regionserver
>Affects Versions: 2.0.0, 1.3.0
>Reporter: Yu Li
>Assignee: Yu Li
> Fix For: 2.0.0, 1.3.0
>
> Attachments: HBASE-14969.branch-1.patch, HBASE-14969.patch,
> HBASE-14969_v10.patch, HBASE-14969_v2.patch, HBASE-14969_v3.patch,
> HBASE-14969_v4.patch, HBASE-14969_v5.patch, HBASE-14969_v6.patch,
> HBASE-14969_v9.patch, fd78628_9909808_compat_report.html,
> load-nothrottling.log, load-throttling.log
>
>
> In HBASE-8329 we added a throughput controller for compaction, to avoid spike
> caused by huge IO pressure like network/disk overflow. However, even with
> this control on, we are still observing disk utils near 100%, and by analysis
> we think this is caused by flush, especially when we increase the setting of
> {{hbase.hstore.flusher.count}}
> In this JIRA, we propose to add throughput control feature for flush, as a
> supplement of HBASE-8329 to better control IO pressure.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14969) Add throughput controller for flush
[ https://issues.apache.org/jira/browse/HBASE-14969?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15122845#comment-15122845 ] Hudson commented on HBASE-14969: SUCCESS: Integrated in HBase-1.3-IT #467 (See [https://builds.apache.org/job/HBase-1.3-IT/467/]) HBASE-14969 Add throughput controller for flush (zhangduo: rev 0d21fa92791ae7d704f48311539facba7061770b) * hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/throttle/TestFlushWithThroughputController.java * hbase-server/src/test/java/org/apache/hadoop/hbase/MockRegionServerServices.java * hbase-server/src/test/java/org/apache/hadoop/hbase/coprocessor/TestRegionObserverScannerOpenHook.java * hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/throttle/CompactionThroughputControllerFactory.java * hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestCompactSplitThread.java * hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/throttle/PressureAwareCompactionThroughputController.java * hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/compactions/CompactionThroughputController.java * hbase-server/src/test/java/org/apache/hadoop/hbase/master/MockRegionServer.java * hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java * hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/compactions/PressureAwareCompactionThroughputController.java * hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/compactions/TestStripeCompactionPolicy.java * hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/compactions/NoLimitCompactionThroughputController.java * hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/compactions/TestCompactionWithThroughputController.java * hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/StripeStoreEngine.java * hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/compactions/StripeCompactor.java * hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/CompactionTool.java * hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/StripeStoreFlusher.java * hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/compactions/CompactionThroughputControllerFactory.java * hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/CompactSplitThread.java * hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestStripeCompactor.java * hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/compactions/CompactionContext.java * hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/throttle/ThroughputController.java * hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/throttle/PressureAwareThroughputController.java * hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/DefaultStoreFlusher.java * hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HStore.java * hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/compactions/DefaultCompactor.java * hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java * hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/DefaultStoreEngine.java * hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/throttle/ThroughputControlUtil.java * hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/throttle/TestCompactionWithThroughputController.java * hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestStripeStoreEngine.java * hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestCompaction.java * hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/compactions/StripeCompactionPolicy.java * hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestSplitTransactionOnCluster.java * hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestWALReplay.java * hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestStore.java * hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/StoreFlusher.java * hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/Store.java * hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestHRegionReplayEvents.java * hbase-server/src/test/java/org/apache/hadoop/hbase/TestIOFencing.java * hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/throttle/NoLimitThroughputController.java * hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/RegionServerServices.java * hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/throttle/PressureAwareFlushThroughputController.java * hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/throttle/FlushThroughputControllerFactory.java * hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/compactions/Compactor.java > Add throughput controller for flush > --- >
[jira] [Commented] (HBASE-14969) Add throughput controller for flush
[
https://issues.apache.org/jira/browse/HBASE-14969?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15116786#comment-15116786
]
Duo Zhang commented on HBASE-14969:
---
OK, Let's finish this...
Will push to master and branch-1 later if no objections.
> Add throughput controller for flush
> ---
>
> Key: HBASE-14969
> URL: https://issues.apache.org/jira/browse/HBASE-14969
> Project: HBase
> Issue Type: Improvement
>Affects Versions: 2.0.0
>Reporter: Yu Li
>Assignee: Yu Li
> Fix For: 2.0.0, 1.3.0
>
> Attachments: HBASE-14969.branch-1.patch, HBASE-14969.patch,
> HBASE-14969_v10.patch, HBASE-14969_v2.patch, HBASE-14969_v3.patch,
> HBASE-14969_v4.patch, HBASE-14969_v5.patch, HBASE-14969_v6.patch,
> HBASE-14969_v9.patch, fd78628_9909808_compat_report.html,
> load-nothrottling.log, load-throttling.log
>
>
> In HBASE-8329 we added a throughput controller for compaction, to avoid spike
> caused by huge IO pressure like network/disk overflow. However, even with
> this control on, we are still observing disk utils near 100%, and by analysis
> we think this is caused by flush, especially when we increase the setting of
> {{hbase.hstore.flusher.count}}
> In this JIRA, we propose to add throughput control feature for flush, as a
> supplement of HBASE-8329 to better control IO pressure.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14969) Add throughput controller for flush
[
https://issues.apache.org/jira/browse/HBASE-14969?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15112243#comment-15112243
]
Yu Li commented on HBASE-14969:
---
Checking UT failures:
* TestImportExport.testDurability:
{noformat}
java.util.concurrent.ExecutionException: java.lang.RuntimeException: Error
while running command to get file permissions :
ExitCodeException exitCode=127: /bin/ls: error while loading shared libraries:
libc.so.6: failed to map segment from shared object:
Permission denied
{noformat}
* TestImportExport.testWithMultipleDeleteFamilyMarkersOfSameRowSameFamily:
{noformat}
java.io.IOException: java.util.concurrent.ExecutionException: ExitCodeException
exitCode=137:
at java.util.concurrent.FutureTask.report(FutureTask.java:122)
at java.util.concurrent.FutureTask.get(FutureTask.java:188)
{noformat}
Both have nothing to do with changes here.
> Add throughput controller for flush
> ---
>
> Key: HBASE-14969
> URL: https://issues.apache.org/jira/browse/HBASE-14969
> Project: HBase
> Issue Type: Improvement
>Affects Versions: 2.0.0
>Reporter: Yu Li
>Assignee: Yu Li
> Fix For: 2.0.0, 1.3.0
>
> Attachments: HBASE-14969.branch-1.patch, HBASE-14969.patch,
> HBASE-14969_v10.patch, HBASE-14969_v2.patch, HBASE-14969_v3.patch,
> HBASE-14969_v4.patch, HBASE-14969_v5.patch, HBASE-14969_v6.patch,
> HBASE-14969_v9.patch, fd78628_9909808_compat_report.html,
> load-nothrottling.log, load-throttling.log
>
>
> In HBASE-8329 we added a throughput controller for compaction, to avoid spike
> caused by huge IO pressure like network/disk overflow. However, even with
> this control on, we are still observing disk utils near 100%, and by analysis
> we think this is caused by flush, especially when we increase the setting of
> {{hbase.hstore.flusher.count}}
> In this JIRA, we propose to add throughput control feature for flush, as a
> supplement of HBASE-8329 to better control IO pressure.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14969) Add throughput controller for flush
[
https://issues.apache.org/jira/browse/HBASE-14969?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=1511#comment-1511
]
Hadoop QA commented on HBASE-14969:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 0s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 16 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
48s {color} | {color:green} branch-1 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 29s
{color} | {color:green} branch-1 passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 32s
{color} | {color:green} branch-1 passed with JDK v1.7.0_91 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
20s {color} | {color:green} branch-1 passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
18s {color} | {color:green} branch-1 passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
49s {color} | {color:green} branch-1 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 28s
{color} | {color:green} branch-1 passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 32s
{color} | {color:green} branch-1 passed with JDK v1.7.0_91 {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
45s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 29s
{color} | {color:green} the patch passed with JDK v1.8.0_66 {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 4m 8s {color}
| {color:red} hbase-server-jdk1.8.0_66 with JDK v1.8.0_66 generated 6 new
issues (was 6, now 6). {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 29s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 32s
{color} | {color:green} the patch passed with JDK v1.7.0_91 {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 4m 40s {color}
| {color:red} hbase-server-jdk1.7.0_91 with JDK v1.7.0_91 generated 6 new
issues (was 6, now 6). {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 32s
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m 15s
{color} | {color:red} Patch generated 6 new checkstyle issues in hbase-server
(total was 159, now 164). {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
16s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
1s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green} 4m
23s {color} | {color:green} Patch does not cause any errors with Hadoop 2.4.1
2.5.2 2.6.0. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
56s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 24s
{color} | {color:green} the patch passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 31s
{color} | {color:green} the patch passed with JDK v1.7.0_91 {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 82m 46s
{color} | {color:green} hbase-server in the patch passed with JDK v1.8.0_66.
{color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 87m 31s {color}
| {color:red} hbase-server in the patch failed with JDK v1.7.0_91. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
17s {color} | {color:green} Patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 199m 41s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| JDK v1.7.0_91 Failed junit tests | hadoop.hbase.mapreduce.TestImportExport |
\\
\\
|| Subsystem || Report/Notes ||
|
[jira] [Commented] (HBASE-14969) Add throughput controller for flush
[
https://issues.apache.org/jira/browse/HBASE-14969?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15112438#comment-15112438
]
Duo Zhang commented on HBASE-14969:
---
[~apurtell] In this patch we move and remove some classes marked as
{{IA.CONFIG}} and map the old class name to new class name in Factory
class(which means we are still compatible with old config file). Does this
break our compatibility assumptions? Thanks.
> Add throughput controller for flush
> ---
>
> Key: HBASE-14969
> URL: https://issues.apache.org/jira/browse/HBASE-14969
> Project: HBase
> Issue Type: Improvement
>Affects Versions: 2.0.0
>Reporter: Yu Li
>Assignee: Yu Li
> Fix For: 2.0.0, 1.3.0
>
> Attachments: HBASE-14969.branch-1.patch, HBASE-14969.patch,
> HBASE-14969_v10.patch, HBASE-14969_v2.patch, HBASE-14969_v3.patch,
> HBASE-14969_v4.patch, HBASE-14969_v5.patch, HBASE-14969_v6.patch,
> HBASE-14969_v9.patch, fd78628_9909808_compat_report.html,
> load-nothrottling.log, load-throttling.log
>
>
> In HBASE-8329 we added a throughput controller for compaction, to avoid spike
> caused by huge IO pressure like network/disk overflow. However, even with
> this control on, we are still observing disk utils near 100%, and by analysis
> we think this is caused by flush, especially when we increase the setting of
> {{hbase.hstore.flusher.count}}
> In this JIRA, we propose to add throughput control feature for flush, as a
> supplement of HBASE-8329 to better control IO pressure.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14969) Add throughput controller for flush
[
https://issues.apache.org/jira/browse/HBASE-14969?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15105285#comment-15105285
]
Yu Li commented on HBASE-14969:
---
The two findbugs issues don't need to handle as mentioned before.
The hadoopcheck issue is tracked by HBASE-15103, we could see the same error in
log:
{noformat}
[ERROR] Error invoking method 'get(java.lang.Integer)' in java.util.ArrayList
at META-INF/LICENSE.vm[line 1627, column 22]
java.lang.reflect.InvocationTargetException
at sun.reflect.GeneratedMethodAccessor139.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at
org.apache.velocity.util.introspection.UberspectImpl$VelMethodImpl.doInvoke(UberspectImpl.java:395)
at
org.apache.velocity.util.introspection.UberspectImpl$VelMethodImpl.invoke(UberspectImpl.java:384)
at
org.apache.velocity.runtime.parser.node.ASTIndex.execute(ASTIndex.java:149)
{noformat}
Will update the patch to resolve new checkstyle issues.
> Add throughput controller for flush
> ---
>
> Key: HBASE-14969
> URL: https://issues.apache.org/jira/browse/HBASE-14969
> Project: HBase
> Issue Type: Improvement
>Affects Versions: 2.0.0
>Reporter: Yu Li
>Assignee: Yu Li
> Fix For: 2.0.0, 1.3.0
>
> Attachments: HBASE-14969.patch, HBASE-14969_v2.patch,
> HBASE-14969_v3.patch, HBASE-14969_v4.patch, HBASE-14969_v5.patch,
> HBASE-14969_v6.patch, HBASE-14969_v9.patch, load-nothrottling.log,
> load-throttling.log
>
>
> In HBASE-8329 we added a throughput controller for compaction, to avoid spike
> caused by huge IO pressure like network/disk overflow. However, even with
> this control on, we are still observing disk utils near 100%, and by analysis
> we think this is caused by flush, especially when we increase the setting of
> {{hbase.hstore.flusher.count}}
> In this JIRA, we propose to add throughput control feature for flush, as a
> supplement of HBASE-8329 to better control IO pressure.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14969) Add throughput controller for flush
[
https://issues.apache.org/jira/browse/HBASE-14969?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15105036#comment-15105036
]
Hadoop QA commented on HBASE-14969:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 17 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
38s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 44s
{color} | {color:green} master passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 38s
{color} | {color:green} master passed with JDK v1.7.0_79 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 4m
40s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
16s {color} | {color:green} master passed {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 2m 22s
{color} | {color:red} hbase-server in master has 82 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 34s
{color} | {color:green} master passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 39s
{color} | {color:green} master passed with JDK v1.7.0_79 {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
53s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 47s
{color} | {color:green} the patch passed with JDK v1.8.0 {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 5m 18s {color}
| {color:red} hbase-server-jdk1.8.0 with JDK v1.8.0 generated 6 new issues (was
6, now 6). {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 47s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 36s
{color} | {color:green} the patch passed with JDK v1.7.0_79 {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 5m 54s {color}
| {color:red} hbase-server-jdk1.7.0_79 with JDK v1.7.0_79 generated 6 new
issues (was 6, now 6). {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 36s
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 4m 21s
{color} | {color:red} Patch generated 2 new checkstyle issues in hbase-server
(total was 568, now 563). {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
14s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:red}-1{color} | {color:red} hadoopcheck {color} | {color:red} 1m 18s
{color} | {color:red} Patch causes 11 errors with Hadoop v2.4.0. {color} |
| {color:red}-1{color} | {color:red} hadoopcheck {color} | {color:red} 2m 37s
{color} | {color:red} Patch causes 11 errors with Hadoop v2.4.1. {color} |
| {color:red}-1{color} | {color:red} hadoopcheck {color} | {color:red} 3m 54s
{color} | {color:red} Patch causes 11 errors with Hadoop v2.5.0. {color} |
| {color:red}-1{color} | {color:red} hadoopcheck {color} | {color:red} 5m 13s
{color} | {color:red} Patch causes 11 errors with Hadoop v2.5.1. {color} |
| {color:red}-1{color} | {color:red} hadoopcheck {color} | {color:red} 6m 31s
{color} | {color:red} Patch causes 11 errors with Hadoop v2.5.2. {color} |
| {color:red}-1{color} | {color:red} hadoopcheck {color} | {color:red} 7m 50s
{color} | {color:red} Patch causes 11 errors with Hadoop v2.6.1. {color} |
| {color:red}-1{color} | {color:red} hadoopcheck {color} | {color:red} 9m 10s
{color} | {color:red} Patch causes 11 errors with Hadoop v2.6.2. {color} |
| {color:red}-1{color} | {color:red} hadoopcheck {color} | {color:red} 10m 30s
{color} | {color:red} Patch causes 11 errors with Hadoop v2.6.3. {color} |
| {color:red}-1{color} | {color:red} hadoopcheck {color} | {color:red} 11m 49s
{color} | {color:red} Patch causes 11 errors with Hadoop v2.7.1. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 2m 1s
{color} | {color:red}
[jira] [Commented] (HBASE-14969) Add throughput controller for flush
[
https://issues.apache.org/jira/browse/HBASE-14969?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15105566#comment-15105566
]
Yu Li commented on HBASE-14969:
---
The latest HadoopQA run looks good.
> Add throughput controller for flush
> ---
>
> Key: HBASE-14969
> URL: https://issues.apache.org/jira/browse/HBASE-14969
> Project: HBase
> Issue Type: Improvement
>Affects Versions: 2.0.0
>Reporter: Yu Li
>Assignee: Yu Li
> Fix For: 2.0.0, 1.3.0
>
> Attachments: HBASE-14969.patch, HBASE-14969_v10.patch,
> HBASE-14969_v2.patch, HBASE-14969_v3.patch, HBASE-14969_v4.patch,
> HBASE-14969_v5.patch, HBASE-14969_v6.patch, HBASE-14969_v9.patch,
> load-nothrottling.log, load-throttling.log
>
>
> In HBASE-8329 we added a throughput controller for compaction, to avoid spike
> caused by huge IO pressure like network/disk overflow. However, even with
> this control on, we are still observing disk utils near 100%, and by analysis
> we think this is caused by flush, especially when we increase the setting of
> {{hbase.hstore.flusher.count}}
> In this JIRA, we propose to add throughput control feature for flush, as a
> supplement of HBASE-8329 to better control IO pressure.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14969) Add throughput controller for flush
[
https://issues.apache.org/jira/browse/HBASE-14969?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15105521#comment-15105521
]
Hadoop QA commented on HBASE-14969:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 17 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
15s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 36s
{color} | {color:green} master passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 31s
{color} | {color:green} master passed with JDK v1.7.0_79 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 4m
10s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
14s {color} | {color:green} master passed {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m 49s
{color} | {color:red} hbase-server in master has 82 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 24s
{color} | {color:green} master passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 31s
{color} | {color:green} master passed with JDK v1.7.0_79 {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
44s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 44s
{color} | {color:green} the patch passed with JDK v1.8.0 {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 4m 16s {color}
| {color:red} hbase-server-jdk1.8.0 with JDK v1.8.0 generated 6 new issues (was
6, now 6). {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 44s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 31s
{color} | {color:green} the patch passed with JDK v1.7.0_79 {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 4m 47s {color}
| {color:red} hbase-server-jdk1.7.0_79 with JDK v1.7.0_79 generated 6 new
issues (was 6, now 6). {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 31s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 4m
22s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
15s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:red}-1{color} | {color:red} hadoopcheck {color} | {color:red} 1m 19s
{color} | {color:red} Patch causes 11 errors with Hadoop v2.4.0. {color} |
| {color:red}-1{color} | {color:red} hadoopcheck {color} | {color:red} 2m 38s
{color} | {color:red} Patch causes 11 errors with Hadoop v2.4.1. {color} |
| {color:red}-1{color} | {color:red} hadoopcheck {color} | {color:red} 3m 57s
{color} | {color:red} Patch causes 11 errors with Hadoop v2.5.0. {color} |
| {color:red}-1{color} | {color:red} hadoopcheck {color} | {color:red} 5m 15s
{color} | {color:red} Patch causes 11 errors with Hadoop v2.5.1. {color} |
| {color:red}-1{color} | {color:red} hadoopcheck {color} | {color:red} 6m 35s
{color} | {color:red} Patch causes 11 errors with Hadoop v2.5.2. {color} |
| {color:red}-1{color} | {color:red} hadoopcheck {color} | {color:red} 7m 55s
{color} | {color:red} Patch causes 11 errors with Hadoop v2.6.1. {color} |
| {color:red}-1{color} | {color:red} hadoopcheck {color} | {color:red} 9m 15s
{color} | {color:red} Patch causes 11 errors with Hadoop v2.6.2. {color} |
| {color:red}-1{color} | {color:red} hadoopcheck {color} | {color:red} 10m 34s
{color} | {color:red} Patch causes 11 errors with Hadoop v2.6.3. {color} |
| {color:red}-1{color} | {color:red} hadoopcheck {color} | {color:red} 11m 54s
{color} | {color:red} Patch causes 11 errors with Hadoop v2.7.1. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 2m 3s
{color} | {color:red} hbase-server introduced 2 new FindBugs issues. {color} |
|
[jira] [Commented] (HBASE-14969) Add throughput controller for flush
[
https://issues.apache.org/jira/browse/HBASE-14969?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15104134#comment-15104134
]
Yu Li commented on HBASE-14969:
---
Thanks Duo, I've updated the patch in rb according to your review comments and
please help review again.
> Add throughput controller for flush
> ---
>
> Key: HBASE-14969
> URL: https://issues.apache.org/jira/browse/HBASE-14969
> Project: HBase
> Issue Type: Improvement
>Affects Versions: 2.0.0
>Reporter: Yu Li
>Assignee: Yu Li
> Fix For: 2.0.0, 1.3.0
>
> Attachments: HBASE-14969.patch, HBASE-14969_v2.patch,
> HBASE-14969_v3.patch, HBASE-14969_v4.patch, HBASE-14969_v5.patch,
> HBASE-14969_v6.patch, load-nothrottling.log, load-throttling.log
>
>
> In HBASE-8329 we added a throughput controller for compaction, to avoid spike
> caused by huge IO pressure like network/disk overflow. However, even with
> this control on, we are still observing disk utils near 100%, and by analysis
> we think this is caused by flush, especially when we increase the setting of
> {{hbase.hstore.flusher.count}}
> In this JIRA, we propose to add throughput control feature for flush, as a
> supplement of HBASE-8329 to better control IO pressure.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14969) Add throughput controller for flush
[
https://issues.apache.org/jira/browse/HBASE-14969?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15103724#comment-15103724
]
Duo Zhang commented on HBASE-14969:
---
{quote}
My only concern is that this will require user to modify their configuration if
use PressureAwareCompressionThroughputController.
{quote}
Yes this is a problem. I think we could map
{{org.apache.hadoop.hbase.regionserver.compactions.PressureAwareCompressionThroughputController}}
to
{{org.apache.hadoop.hbase.regionserver.throttle.PressureAwareCompressionThroughputController}}
in {{CompactionThroughputControllerFactory}}. And also for
{{NoLimitCompactionThroughputController}}.
{quote}
Is it possible we add any separate logic in setup/start/finish method for flush
and compaction no-limit-controller in the future?
{quote}
I think 'no limit' itself does not have other meanings. People could introduce
other {{ThroughputController}} if they want(just use a different name).
And I left some comments on rb, only small changes.
Thanks.
> Add throughput controller for flush
> ---
>
> Key: HBASE-14969
> URL: https://issues.apache.org/jira/browse/HBASE-14969
> Project: HBase
> Issue Type: Improvement
>Affects Versions: 2.0.0
>Reporter: Yu Li
>Assignee: Yu Li
> Fix For: 2.0.0, 1.3.0
>
> Attachments: HBASE-14969.patch, HBASE-14969_v2.patch,
> HBASE-14969_v3.patch, HBASE-14969_v4.patch, HBASE-14969_v5.patch,
> HBASE-14969_v6.patch, load-nothrottling.log, load-throttling.log
>
>
> In HBASE-8329 we added a throughput controller for compaction, to avoid spike
> caused by huge IO pressure like network/disk overflow. However, even with
> this control on, we are still observing disk utils near 100%, and by analysis
> we think this is caused by flush, especially when we increase the setting of
> {{hbase.hstore.flusher.count}}
> In this JIRA, we propose to add throughput control feature for flush, as a
> supplement of HBASE-8329 to better control IO pressure.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14969) Add throughput controller for flush
[
https://issues.apache.org/jira/browse/HBASE-14969?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15103274#comment-15103274
]
Yu Li commented on HBASE-14969:
---
Thanks for the review [~Apache9]
bq. upload the newest patch to review board
Just done.
bq. I suggest we also move compaction related controller classes into that
package
My only concern is that this will require user to modify their configuration if
use PressureAwareCompressionThroughputController. Please let me know your
thoughts here.
bq. And do we need two different {{NoLimitThroughputController}} for flush and
compaction?
Is it possible we add any separate logic in setup/start/finish method for flush
and compaction no-limit-controller in the future? If not, I agree we only
reserve NoLimitThroughputController and remove the other two.
> Add throughput controller for flush
> ---
>
> Key: HBASE-14969
> URL: https://issues.apache.org/jira/browse/HBASE-14969
> Project: HBase
> Issue Type: Improvement
>Affects Versions: 2.0.0
>Reporter: Yu Li
>Assignee: Yu Li
> Fix For: 2.0.0, 1.3.0
>
> Attachments: HBASE-14969.patch, HBASE-14969_v2.patch,
> HBASE-14969_v3.patch, HBASE-14969_v4.patch, HBASE-14969_v5.patch,
> HBASE-14969_v6.patch, load-nothrottling.log, load-throttling.log
>
>
> In HBASE-8329 we added a throughput controller for compaction, to avoid spike
> caused by huge IO pressure like network/disk overflow. However, even with
> this control on, we are still observing disk utils near 100%, and by analysis
> we think this is caused by flush, especially when we increase the setting of
> {{hbase.hstore.flusher.count}}
> In this JIRA, we propose to add throughput control feature for flush, as a
> supplement of HBASE-8329 to better control IO pressure.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14969) Add throughput controller for flush
[
https://issues.apache.org/jira/browse/HBASE-14969?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15101868#comment-15101868
]
Yu Li commented on HBASE-14969:
---
Checked and confirmed the UT failures and findbugs issues are not introduced by
the latest patch, and no more checkstyle issues.
> Add throughput controller for flush
> ---
>
> Key: HBASE-14969
> URL: https://issues.apache.org/jira/browse/HBASE-14969
> Project: HBase
> Issue Type: Improvement
>Affects Versions: 2.0.0
>Reporter: Yu Li
>Assignee: Yu Li
> Fix For: 2.0.0, 1.3.0
>
> Attachments: HBASE-14969.patch, HBASE-14969_v2.patch,
> HBASE-14969_v3.patch, HBASE-14969_v4.patch, HBASE-14969_v5.patch,
> HBASE-14969_v6.patch, load-nothrottling.log, load-throttling.log
>
>
> In HBASE-8329 we added a throughput controller for compaction, to avoid spike
> caused by huge IO pressure like network/disk overflow. However, even with
> this control on, we are still observing disk utils near 100%, and by analysis
> we think this is caused by flush, especially when we increase the setting of
> {{hbase.hstore.flusher.count}}
> In this JIRA, we propose to add throughput control feature for flush, as a
> supplement of HBASE-8329 to better control IO pressure.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14969) Add throughput controller for flush
[
https://issues.apache.org/jira/browse/HBASE-14969?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15101822#comment-15101822
]
Hadoop QA commented on HBASE-14969:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 9 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
24s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 53s
{color} | {color:green} master passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 38s
{color} | {color:green} master passed with JDK v1.7.0_79 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 4m
44s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
14s {color} | {color:green} master passed {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 2m 3s
{color} | {color:red} hbase-server in master has 83 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 34s
{color} | {color:green} master passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 34s
{color} | {color:green} master passed with JDK v1.7.0_79 {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
41s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 2s
{color} | {color:green} the patch passed with JDK v1.8.0 {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 4m 59s {color}
| {color:red} hbase-server-jdk1.8.0 with JDK v1.8.0 generated 6 new issues (was
6, now 6). {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m 2s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 31s
{color} | {color:green} the patch passed with JDK v1.7.0_79 {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 5m 30s {color}
| {color:red} hbase-server-jdk1.7.0_79 with JDK v1.7.0_79 generated 6 new
issues (was 6, now 6). {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 31s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 3m
57s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
15s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
21m 42s {color} | {color:green} Patch does not cause any errors with Hadoop
2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.1 2.6.2 2.6.3 2.7.1. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 2m 9s
{color} | {color:red} hbase-server introduced 3 new FindBugs issues. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 35s
{color} | {color:green} the patch passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 35s
{color} | {color:green} the patch passed with JDK v1.7.0_79 {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 113m 48s
{color} | {color:red} hbase-server in the patch failed with JDK v1.8.0. {color}
|
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 105m 31s
{color} | {color:green} hbase-server in the patch passed with JDK v1.7.0_79.
{color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
18s {color} | {color:green} Patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 263m 37s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| FindBugs | module:hbase-server |
| | Increment of volatile field
org.apache.hadoop.hbase.regionserver.HRegion$WriteState.compacting in
org.apache.hadoop.hbase.regionserver.HRegion.compact(CompactionContext, Store,
ThroughputController, User) At
[jira] [Commented] (HBASE-14969) Add throughput controller for flush
[
https://issues.apache.org/jira/browse/HBASE-14969?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15102063#comment-15102063
]
Ted Yu commented on HBASE-14969:
lgtm
> Add throughput controller for flush
> ---
>
> Key: HBASE-14969
> URL: https://issues.apache.org/jira/browse/HBASE-14969
> Project: HBase
> Issue Type: Improvement
>Affects Versions: 2.0.0
>Reporter: Yu Li
>Assignee: Yu Li
> Fix For: 2.0.0, 1.3.0
>
> Attachments: HBASE-14969.patch, HBASE-14969_v2.patch,
> HBASE-14969_v3.patch, HBASE-14969_v4.patch, HBASE-14969_v5.patch,
> HBASE-14969_v6.patch, load-nothrottling.log, load-throttling.log
>
>
> In HBASE-8329 we added a throughput controller for compaction, to avoid spike
> caused by huge IO pressure like network/disk overflow. However, even with
> this control on, we are still observing disk utils near 100%, and by analysis
> we think this is caused by flush, especially when we increase the setting of
> {{hbase.hstore.flusher.count}}
> In this JIRA, we propose to add throughput control feature for flush, as a
> supplement of HBASE-8329 to better control IO pressure.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14969) Add throughput controller for flush
[
https://issues.apache.org/jira/browse/HBASE-14969?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15102910#comment-15102910
]
Duo Zhang commented on HBASE-14969:
---
In general, since we have a new package 'controller' and flush related
controller classes are put in that package, I suggest we also move compaction
related controller classes into that package. And I think the package name
should be ‘throttle’? The reason I chose the name 'ThroughputController' is
that there is already a 'throttle' when selecting thread pool when doing
compaction. But 'controller' usually leads people to the scope of MVC, so I
think we should use 'throttle' here.
And do we need two different {{NoLimitThroughputController}} for flush and
compaction? I do not think it is worth introducing two classes with only
different {{toString}} methods...
The whole patch looks good to me, no big concerns.
Thanks.
> Add throughput controller for flush
> ---
>
> Key: HBASE-14969
> URL: https://issues.apache.org/jira/browse/HBASE-14969
> Project: HBase
> Issue Type: Improvement
>Affects Versions: 2.0.0
>Reporter: Yu Li
>Assignee: Yu Li
> Fix For: 2.0.0, 1.3.0
>
> Attachments: HBASE-14969.patch, HBASE-14969_v2.patch,
> HBASE-14969_v3.patch, HBASE-14969_v4.patch, HBASE-14969_v5.patch,
> HBASE-14969_v6.patch, load-nothrottling.log, load-throttling.log
>
>
> In HBASE-8329 we added a throughput controller for compaction, to avoid spike
> caused by huge IO pressure like network/disk overflow. However, even with
> this control on, we are still observing disk utils near 100%, and by analysis
> we think this is caused by flush, especially when we increase the setting of
> {{hbase.hstore.flusher.count}}
> In this JIRA, we propose to add throughput control feature for flush, as a
> supplement of HBASE-8329 to better control IO pressure.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14969) Add throughput controller for flush
[
https://issues.apache.org/jira/browse/HBASE-14969?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15102889#comment-15102889
]
Duo Zhang commented on HBASE-14969:
---
[~liyu] Could you please upload the newest patch to review board? Thanks.
> Add throughput controller for flush
> ---
>
> Key: HBASE-14969
> URL: https://issues.apache.org/jira/browse/HBASE-14969
> Project: HBase
> Issue Type: Improvement
>Affects Versions: 2.0.0
>Reporter: Yu Li
>Assignee: Yu Li
> Fix For: 2.0.0, 1.3.0
>
> Attachments: HBASE-14969.patch, HBASE-14969_v2.patch,
> HBASE-14969_v3.patch, HBASE-14969_v4.patch, HBASE-14969_v5.patch,
> HBASE-14969_v6.patch, load-nothrottling.log, load-throttling.log
>
>
> In HBASE-8329 we added a throughput controller for compaction, to avoid spike
> caused by huge IO pressure like network/disk overflow. However, even with
> this control on, we are still observing disk utils near 100%, and by analysis
> we think this is caused by flush, especially when we increase the setting of
> {{hbase.hstore.flusher.count}}
> In this JIRA, we propose to add throughput control feature for flush, as a
> supplement of HBASE-8329 to better control IO pressure.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14969) Add throughput controller for flush
[
https://issues.apache.org/jira/browse/HBASE-14969?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15101427#comment-15101427
]
Hadoop QA commented on HBASE-14969:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 9 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
15s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 45s
{color} | {color:green} master passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 31s
{color} | {color:green} master passed with JDK v1.7.0_79 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 4m
20s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
15s {color} | {color:green} master passed {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m 47s
{color} | {color:red} hbase-server in master has 83 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 24s
{color} | {color:green} master passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 30s
{color} | {color:green} master passed with JDK v1.7.0_79 {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
42s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 44s
{color} | {color:green} the patch passed with JDK v1.8.0 {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 4m 10s {color}
| {color:red} hbase-server-jdk1.8.0 with JDK v1.8.0 generated 6 new issues (was
6, now 6). {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 44s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 31s
{color} | {color:green} the patch passed with JDK v1.7.0_79 {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 4m 41s {color}
| {color:red} hbase-server-jdk1.7.0_79 with JDK v1.7.0_79 generated 6 new
issues (was 6, now 6). {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 31s
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 4m 6s
{color} | {color:red} Patch generated 2 new checkstyle issues in hbase-server
(total was 556, now 551). {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
15s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
20m 10s {color} | {color:green} Patch does not cause any errors with Hadoop
2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.1 2.6.2 2.6.3 2.7.1. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 2m 2s
{color} | {color:red} hbase-server introduced 3 new FindBugs issues. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 24s
{color} | {color:green} the patch passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 29s
{color} | {color:green} the patch passed with JDK v1.7.0_79 {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 100m 47s
{color} | {color:red} hbase-server in the patch failed with JDK v1.8.0. {color}
|
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 0m 18s {color}
| {color:red} hbase-server in the patch failed with JDK v1.7.0_79. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
9s {color} | {color:green} Patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 141m 48s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| FindBugs | module:hbase-server |
| | Increment of volatile field
org.apache.hadoop.hbase.regionserver.HRegion$WriteState.compacting in
[jira] [Commented] (HBASE-14969) Add throughput controller for flush
[
https://issues.apache.org/jira/browse/HBASE-14969?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15101436#comment-15101436
]
Hadoop QA commented on HBASE-14969:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 9 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
58s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 47s
{color} | {color:green} master passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 31s
{color} | {color:green} master passed with JDK v1.7.0_79 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 4m
14s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
19s {color} | {color:green} master passed {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m 51s
{color} | {color:red} hbase-server in master has 83 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 32s
{color} | {color:green} master passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 33s
{color} | {color:green} master passed with JDK v1.7.0_79 {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
43s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 41s
{color} | {color:green} the patch passed with JDK v1.8.0 {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 4m 24s {color}
| {color:red} hbase-server-jdk1.8.0 with JDK v1.8.0 generated 6 new issues (was
6, now 6). {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 41s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 33s
{color} | {color:green} the patch passed with JDK v1.7.0_79 {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 4m 57s {color}
| {color:red} hbase-server-jdk1.7.0_79 with JDK v1.7.0_79 generated 6 new
issues (was 6, now 6). {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 33s
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 4m 11s
{color} | {color:red} Patch generated 2 new checkstyle issues in hbase-server
(total was 556, now 551). {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
15s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
25m 24s {color} | {color:green} Patch does not cause any errors with Hadoop
2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.1 2.6.2 2.6.3 2.7.1. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 2m 41s
{color} | {color:red} hbase-server introduced 3 new FindBugs issues. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 40s
{color} | {color:green} the patch passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 40s
{color} | {color:green} the patch passed with JDK v1.7.0_79 {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 113m 27s
{color} | {color:green} hbase-server in the patch passed with JDK v1.8.0.
{color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 19m 55s {color}
| {color:red} hbase-server in the patch failed with JDK v1.7.0_79. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
15s {color} | {color:green} Patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 182m 36s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| FindBugs | module:hbase-server |
| | Increment of volatile field
org.apache.hadoop.hbase.regionserver.HRegion$WriteState.compacting in
[jira] [Commented] (HBASE-14969) Add throughput controller for flush
[
https://issues.apache.org/jira/browse/HBASE-14969?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15101549#comment-15101549
]
Yu Li commented on HBASE-14969:
---
Checking the UT result: the failure is on TestAsyncSecureIPC and caused by
java.net.BindException, it's irrelative to this patch
There're still 2 left over checkstyle issues
{noformat}
error: Class CompactionThroughputControllerFactory should be declared as final.
error: Class FlushThroughputControllerFactory should be declared as final.
{noformat}
Will upload another patch to fix them
> Add throughput controller for flush
> ---
>
> Key: HBASE-14969
> URL: https://issues.apache.org/jira/browse/HBASE-14969
> Project: HBase
> Issue Type: Improvement
>Affects Versions: 2.0.0
>Reporter: Yu Li
>Assignee: Yu Li
> Fix For: 2.0.0, 1.3.0
>
> Attachments: HBASE-14969.patch, HBASE-14969_v2.patch,
> HBASE-14969_v3.patch, HBASE-14969_v4.patch, HBASE-14969_v5.patch,
> load-nothrottling.log, load-throttling.log
>
>
> In HBASE-8329 we added a throughput controller for compaction, to avoid spike
> caused by huge IO pressure like network/disk overflow. However, even with
> this control on, we are still observing disk utils near 100%, and by analysis
> we think this is caused by flush, especially when we increase the setting of
> {{hbase.hstore.flusher.count}}
> In this JIRA, we propose to add throughput control feature for flush, as a
> supplement of HBASE-8329 to better control IO pressure.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14969) Add throughput controller for flush
[
https://issues.apache.org/jira/browse/HBASE-14969?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15098530#comment-15098530
]
Ted Yu commented on HBASE-14969:
Performance numbers above are impressive.
> Add throughput controller for flush
> ---
>
> Key: HBASE-14969
> URL: https://issues.apache.org/jira/browse/HBASE-14969
> Project: HBase
> Issue Type: Improvement
>Affects Versions: 2.0.0
>Reporter: Yu Li
>Assignee: Yu Li
> Fix For: 2.0.0, 1.3.0
>
> Attachments: HBASE-14969.patch, HBASE-14969_v2.patch,
> HBASE-14969_v3.patch
>
>
> In HBASE-8329 we added a throughput controller for compaction, to avoid spike
> caused by huge IO pressure like network/disk overflow. However, even with
> this control on, we are still observing disk utils near 100%, and by analysis
> we think this is caused by flush, especially when we increase the setting of
> {{hbase.hstore.flusher.count}}
> In this JIRA, we propose to add throughput control feature for flush, as a
> supplement of HBASE-8329 to better control IO pressure.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14969) Add throughput controller for flush
[
https://issues.apache.org/jira/browse/HBASE-14969?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15098476#comment-15098476
]
Yu Li commented on HBASE-14969:
---
[~tedyu] and [~Apache9],
Sorry for the lag, here are the testing results:
Test env (to simulate high flush pressure by ingesting 100GB data into a single
RS cluster):
* 1 single RegionServer, 4 datanodes
* HBase version: commit c8b9754a5e, w/o and w/ patch here
* Test command: {{bin/hbase org.apache.hadoop.hbase.PerformanceEvaluation
--nomapred --presplit=16 --rows=10485760 randomWrite 10}}
Overall time (Average of 10 threads), milliseconds:
* no-throttling: 2909600
* throttling: 2876955
Latency (Average of 10 threads), microseconds:
||Flush-mode||Max||99.999th||99.9th||75th||50th||Min||Avg||StdDev
|no-throttling|1818356.5|1124412.62|21.8999|3.0|2.2|2.0|273.25|11455.52|
|throttling|1467822.8|1084914.03|22.0998|3.0|2.0|2.0|273.12|11590.58|
Notice that we've observed much higher disk utils causing bigger spike in our
online cluster during flush than the testing env, so I think this feature will
be more useful on real environment.
> Add throughput controller for flush
> ---
>
> Key: HBASE-14969
> URL: https://issues.apache.org/jira/browse/HBASE-14969
> Project: HBase
> Issue Type: Improvement
>Affects Versions: 2.0.0
>Reporter: Yu Li
>Assignee: Yu Li
> Attachments: HBASE-14969.patch, HBASE-14969_v2.patch
>
>
> In HBASE-8329 we added a throughput controller for compaction, to avoid spike
> caused by huge IO pressure like network/disk overflow. However, even with
> this control on, we are still observing disk utils near 100%, and by analysis
> we think this is caused by flush, especially when we increase the setting of
> {{hbase.hstore.flusher.count}}
> In this JIRA, we propose to add throughput control feature for flush, as a
> supplement of HBASE-8329 to better control IO pressure.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14969) Add throughput controller for flush
[
https://issues.apache.org/jira/browse/HBASE-14969?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15101144#comment-15101144
]
Ted Yu commented on HBASE-14969:
Can you fix the new checkstyle warnings ?
> Add throughput controller for flush
> ---
>
> Key: HBASE-14969
> URL: https://issues.apache.org/jira/browse/HBASE-14969
> Project: HBase
> Issue Type: Improvement
>Affects Versions: 2.0.0
>Reporter: Yu Li
>Assignee: Yu Li
> Fix For: 2.0.0, 1.3.0
>
> Attachments: HBASE-14969.patch, HBASE-14969_v2.patch,
> HBASE-14969_v3.patch
>
>
> In HBASE-8329 we added a throughput controller for compaction, to avoid spike
> caused by huge IO pressure like network/disk overflow. However, even with
> this control on, we are still observing disk utils near 100%, and by analysis
> we think this is caused by flush, especially when we increase the setting of
> {{hbase.hstore.flusher.count}}
> In this JIRA, we propose to add throughput control feature for flush, as a
> supplement of HBASE-8329 to better control IO pressure.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14969) Add throughput controller for flush
[
https://issues.apache.org/jira/browse/HBASE-14969?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15101141#comment-15101141
]
Yu Li commented on HBASE-14969:
---
Checking the findbugs warning, none is introduced by the patch here.
For the two "Increment of volatile field" warnings in HRegion, since the
increment is inside a synchronized block, it's not a problem.
For the "Result of integer multiplication cast to long" one, have opened
HBASE-15117 to resolve it.
> Add throughput controller for flush
> ---
>
> Key: HBASE-14969
> URL: https://issues.apache.org/jira/browse/HBASE-14969
> Project: HBase
> Issue Type: Improvement
>Affects Versions: 2.0.0
>Reporter: Yu Li
>Assignee: Yu Li
> Fix For: 2.0.0, 1.3.0
>
> Attachments: HBASE-14969.patch, HBASE-14969_v2.patch,
> HBASE-14969_v3.patch
>
>
> In HBASE-8329 we added a throughput controller for compaction, to avoid spike
> caused by huge IO pressure like network/disk overflow. However, even with
> this control on, we are still observing disk utils near 100%, and by analysis
> we think this is caused by flush, especially when we increase the setting of
> {{hbase.hstore.flusher.count}}
> In this JIRA, we propose to add throughput control feature for flush, as a
> supplement of HBASE-8329 to better control IO pressure.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14969) Add throughput controller for flush
[
https://issues.apache.org/jira/browse/HBASE-14969?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15101226#comment-15101226
]
Yu Li commented on HBASE-14969:
---
{quote}
Does the field have to be volatile ?
I don't see synchronization around access to the field.
{quote}
It was marked as volatile and I didn't change this during the refactoring. I
guess this just follows other Stopable implementation like HRegionServer to
make the field volatile. However, based on current logic, I don't think stop
method of ThroughputController will be called in parallel. I'm hesitating
whether to leave it as is or remove the volatile qualifier. What's your opinion
[~tedyu]?
Regarding synchronization, the findbugs warning was in HRegion class not this
NoLimitThroughputController, and I don't think we need synchronization on this
volatile since there's no operation like increment which needs additional atomic
bq. 'is tune' -> 'is tuned'
This is also a copy-paste line... will fix it in the next patch, after got the
confirmation on how to deal with the above volatile filed.
> Add throughput controller for flush
> ---
>
> Key: HBASE-14969
> URL: https://issues.apache.org/jira/browse/HBASE-14969
> Project: HBase
> Issue Type: Improvement
>Affects Versions: 2.0.0
>Reporter: Yu Li
>Assignee: Yu Li
> Fix For: 2.0.0, 1.3.0
>
> Attachments: HBASE-14969.patch, HBASE-14969_v2.patch,
> HBASE-14969_v3.patch, HBASE-14969_v4.patch
>
>
> In HBASE-8329 we added a throughput controller for compaction, to avoid spike
> caused by huge IO pressure like network/disk overflow. However, even with
> this control on, we are still observing disk utils near 100%, and by analysis
> we think this is caused by flush, especially when we increase the setting of
> {{hbase.hstore.flusher.count}}
> In this JIRA, we propose to add throughput control feature for flush, as a
> supplement of HBASE-8329 to better control IO pressure.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14969) Add throughput controller for flush
[
https://issues.apache.org/jira/browse/HBASE-14969?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15101229#comment-15101229
]
Ted Yu commented on HBASE-14969:
I think the volatile can be removed
> Add throughput controller for flush
> ---
>
> Key: HBASE-14969
> URL: https://issues.apache.org/jira/browse/HBASE-14969
> Project: HBase
> Issue Type: Improvement
>Affects Versions: 2.0.0
>Reporter: Yu Li
>Assignee: Yu Li
> Fix For: 2.0.0, 1.3.0
>
> Attachments: HBASE-14969.patch, HBASE-14969_v2.patch,
> HBASE-14969_v3.patch, HBASE-14969_v4.patch
>
>
> In HBASE-8329 we added a throughput controller for compaction, to avoid spike
> caused by huge IO pressure like network/disk overflow. However, even with
> this control on, we are still observing disk utils near 100%, and by analysis
> we think this is caused by flush, especially when we increase the setting of
> {{hbase.hstore.flusher.count}}
> In this JIRA, we propose to add throughput control feature for flush, as a
> supplement of HBASE-8329 to better control IO pressure.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14969) Add throughput controller for flush
[
https://issues.apache.org/jira/browse/HBASE-14969?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15101186#comment-15101186
]
Ted Yu commented on HBASE-14969:
For NoLimitThroughputController :
{code}
44private volatile boolean stopped;
{code}
Does the field have to be volatile ?
I don't see synchronization around access to the field.
{code}
33 * In normal case, the max throughput is tune between
{code}
'is tune' -> 'is tuned'
> Add throughput controller for flush
> ---
>
> Key: HBASE-14969
> URL: https://issues.apache.org/jira/browse/HBASE-14969
> Project: HBase
> Issue Type: Improvement
>Affects Versions: 2.0.0
>Reporter: Yu Li
>Assignee: Yu Li
> Fix For: 2.0.0, 1.3.0
>
> Attachments: HBASE-14969.patch, HBASE-14969_v2.patch,
> HBASE-14969_v3.patch, HBASE-14969_v4.patch
>
>
> In HBASE-8329 we added a throughput controller for compaction, to avoid spike
> caused by huge IO pressure like network/disk overflow. However, even with
> this control on, we are still observing disk utils near 100%, and by analysis
> we think this is caused by flush, especially when we increase the setting of
> {{hbase.hstore.flusher.count}}
> In this JIRA, we propose to add throughput control feature for flush, as a
> supplement of HBASE-8329 to better control IO pressure.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14969) Add throughput controller for flush
[
https://issues.apache.org/jira/browse/HBASE-14969?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15098921#comment-15098921
]
Hadoop QA commented on HBASE-14969:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 9 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
45s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 41s
{color} | {color:green} master passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 34s
{color} | {color:green} master passed with JDK v1.7.0_79 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 4m
19s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
17s {color} | {color:green} master passed {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 2m 4s
{color} | {color:red} hbase-server in master has 83 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 40s
{color} | {color:green} master passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 34s
{color} | {color:green} master passed with JDK v1.7.0_79 {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
44s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 43s
{color} | {color:green} the patch passed with JDK v1.8.0 {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 4m 49s {color}
| {color:red} hbase-server-jdk1.8.0 with JDK v1.8.0 generated 6 new issues (was
6, now 6). {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 43s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 33s
{color} | {color:green} the patch passed with JDK v1.7.0_79 {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 5m 23s {color}
| {color:red} hbase-server-jdk1.7.0_79 with JDK v1.7.0_79 generated 6 new
issues (was 6, now 6). {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 33s
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 4m 34s
{color} | {color:red} Patch generated 5 new checkstyle issues in hbase-server
(total was 556, now 555). {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
16s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
23m 17s {color} | {color:green} Patch does not cause any errors with Hadoop
2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.1 2.6.2 2.6.3 2.7.1. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 2m 12s
{color} | {color:red} hbase-server introduced 3 new FindBugs issues. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 32s
{color} | {color:green} the patch passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 34s
{color} | {color:green} the patch passed with JDK v1.7.0_79 {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 96m 13s
{color} | {color:green} hbase-server in the patch passed with JDK v1.8.0.
{color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 95m 17s
{color} | {color:green} hbase-server in the patch passed with JDK v1.7.0_79.
{color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
28s {color} | {color:green} Patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 237m 42s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| FindBugs | module:hbase-server |
| | Increment of volatile field
org.apache.hadoop.hbase.regionserver.HRegion$WriteState.compacting in
[jira] [Commented] (HBASE-14969) Add throughput controller for flush
[
https://issues.apache.org/jira/browse/HBASE-14969?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15075281#comment-15075281
]
Hadoop QA commented on HBASE-14969:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12779995/HBASE-14969_v2.patch
against master branch at commit 07b623670647686084f8f5fd2038e2bafcfdac54.
ATTACHMENT ID: 12779995
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 26 new
or modified tests.
{color:green}+1 hadoop versions{color}. The patch compiles with all
supported hadoop versions (2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.0 2.6.1 2.7.0
2.7.1)
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 protoc{color}. The applied patch does not increase the
total number of protoc compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:red}-1 checkstyle{color}. The applied patch generated
new checkstyle errors. Check build console for list of new errors.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:green}+1 site{color}. The mvn post-site goal succeeds with this
patch.
{color:green}+1 core tests{color}. The patch passed unit tests in .
{color:green}+1 zombies{color}. No zombie tests found running at the end of
the build.
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/17079//testReport/
Release Findbugs (version 2.0.3)warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/17079//artifact/patchprocess/newFindbugsWarnings.html
Checkstyle Errors:
https://builds.apache.org/job/PreCommit-HBASE-Build/17079//artifact/patchprocess/checkstyle-aggregate.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/17079//console
This message is automatically generated.
> Add throughput controller for flush
> ---
>
> Key: HBASE-14969
> URL: https://issues.apache.org/jira/browse/HBASE-14969
> Project: HBase
> Issue Type: Improvement
>Affects Versions: 2.0.0
>Reporter: Yu Li
>Assignee: Yu Li
> Attachments: HBASE-14969.patch, HBASE-14969_v2.patch
>
>
> In HBASE-8329 we added a throughput controller for compaction, to avoid spike
> caused by huge IO pressure like network/disk overflow. However, even with
> this control on, we are still observing disk utils near 100%, and by analysis
> we think this is caused by flush, especially when we increase the setting of
> {{hbase.hstore.flusher.count}}
> In this JIRA, we propose to add throughput control feature for flush, as a
> supplement of HBASE-8329 to better control IO pressure.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14969) Add throughput controller for flush
[
https://issues.apache.org/jira/browse/HBASE-14969?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15071720#comment-15071720
]
Ted Yu commented on HBASE-14969:
https://builds.apache.org/job/PreCommit-HBASE-Build/16896//testReport/ is no
longer accessible.
Please attach latest patch and see if there is outstanding checkstyle warning.
> Add throughput controller for flush
> ---
>
> Key: HBASE-14969
> URL: https://issues.apache.org/jira/browse/HBASE-14969
> Project: HBase
> Issue Type: Improvement
>Affects Versions: 2.0.0
>Reporter: Yu Li
>Assignee: Yu Li
> Attachments: HBASE-14969.patch
>
>
> In HBASE-8329 we added a throughput controller for compaction, to avoid spike
> caused by huge IO pressure like network/disk overflow. However, even with
> this control on, we are still observing disk utils near 100%, and by analysis
> we think this is caused by flush, especially when we increase the setting of
> {{hbase.hstore.flusher.count}}
> In this JIRA, we propose to add throughput control feature for flush, as a
> supplement of HBASE-8329 to better control IO pressure.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14969) Add throughput controller for flush
[
https://issues.apache.org/jira/browse/HBASE-14969?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15067930#comment-15067930
]
Yu Li commented on HBASE-14969:
---
The javadoc issue is not introduced by this JIRA and already got resolved in
HBASE-15000. Will update the patch to resolve the javacoc/lineLengths issue
after review done on rb.
> Add throughput controller for flush
> ---
>
> Key: HBASE-14969
> URL: https://issues.apache.org/jira/browse/HBASE-14969
> Project: HBase
> Issue Type: Improvement
>Affects Versions: 2.0.0
>Reporter: Yu Li
>Assignee: Yu Li
> Attachments: HBASE-14969.patch
>
>
> In HBASE-8329 we added a throughput controller for compaction, to avoid spike
> caused by huge IO pressure like network/disk overflow. However, even with
> this control on, we are still observing disk utils near 100%, and by analysis
> we think this is caused by flush, especially when we increase the setting of
> {{hbase.hstore.flusher.count}}
> In this JIRA, we propose to add throughput control feature for flush, as a
> supplement of HBASE-8329 to better control IO pressure.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14969) Add throughput controller for flush
[
https://issues.apache.org/jira/browse/HBASE-14969?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15063370#comment-15063370
]
Yu Li commented on HBASE-14969:
---
Thanks [~Apache9] for the follow up.
bq. do you have some perf results already?
I tried to load 100GB data into a 3 nodes dev cluster using PE (--presplit=100
--rows=10485760 randomWrite 10), and could observe less slow sync (threshold
100ms) in the RS log, but didn't see much difference on the latency number in
PE result (similar, no downgrade, but no obvious enhancement also). I guess
it's not easy to reproduce the online situation and will retry. Will paste some
detailed perf number here later.
> Add throughput controller for flush
> ---
>
> Key: HBASE-14969
> URL: https://issues.apache.org/jira/browse/HBASE-14969
> Project: HBase
> Issue Type: Improvement
>Affects Versions: 2.0.0
>Reporter: Yu Li
>Assignee: Yu Li
> Attachments: HBASE-14969.patch
>
>
> In HBASE-8329 we added a throughput controller for compaction, to avoid spike
> caused by huge IO pressure like network/disk overflow. However, even with
> this control on, we are still observing disk utils near 100%, and by analysis
> we think this is caused by flush, especially when we increase the setting of
> {{hbase.hstore.flusher.count}}
> In this JIRA, we propose to add throughput control feature for flush, as a
> supplement of HBASE-8329 to better control IO pressure.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14969) Add throughput controller for flush
[
https://issues.apache.org/jira/browse/HBASE-14969?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15061575#comment-15061575
]
Hadoop QA commented on HBASE-14969:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12777331/HBASE-14969.patch
against master branch at commit 546ac1cbb379acb198b2debabbf14c878ffb8a33.
ATTACHMENT ID: 12777331
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 26 new
or modified tests.
{color:green}+1 hadoop versions{color}. The patch compiles with all
supported hadoop versions (2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.0 2.6.1 2.7.0
2.7.1)
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 protoc{color}. The applied patch does not increase the
total number of protoc compiler warnings.
{color:red}-1 javadoc{color}. The javadoc tool appears to have generated 1
warning messages.
{color:red}-1 checkstyle{color}. The applied patch generated
new checkstyle errors. Check build console for list of new errors.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 lineLengths{color}. The patch introduces the following lines
longer than 100:
+return "DefaultCompactionThroughputController [maxThroughput=" +
throughputDesc(getMaxThroughput())
+ assertTrue(regionServer.getFlushThroughputController() instanceof
NoLimitFlushThroughputController);
{color:green}+1 site{color}. The mvn post-site goal succeeds with this
patch.
{color:green}+1 core tests{color}. The patch passed unit tests in .
{color:green}+1 zombies{color}. No zombie tests found running at the end of
the build.
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/16896//testReport/
Release Findbugs (version 2.0.3)warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/16896//artifact/patchprocess/newFindbugsWarnings.html
Checkstyle Errors:
https://builds.apache.org/job/PreCommit-HBASE-Build/16896//artifact/patchprocess/checkstyle-aggregate.html
Javadoc warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/16896//artifact/patchprocess/patchJavadocWarnings.txt
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/16896//console
This message is automatically generated.
> Add throughput controller for flush
> ---
>
> Key: HBASE-14969
> URL: https://issues.apache.org/jira/browse/HBASE-14969
> Project: HBase
> Issue Type: Improvement
>Affects Versions: 2.0.0
>Reporter: Yu Li
>Assignee: Yu Li
> Attachments: HBASE-14969.patch
>
>
> In HBASE-8329 we added a throughput controller for compaction, to avoid spike
> caused by huge IO pressure like network/disk overflow. However, even with
> this control on, we are still observing disk utils near 100%, and by analysis
> we think this is caused by flush, especially when we increase the setting of
> {{hbase.hstore.flusher.count}}
> In this JIRA, we propose to add throughput control feature for flush, as a
> supplement of HBASE-8329 to better control IO pressure.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14969) Add throughput controller for flush
[
https://issues.apache.org/jira/browse/HBASE-14969?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15061315#comment-15061315
]
Duo Zhang commented on HBASE-14969:
---
Will review the patch later on rb and do you have some perf results already?
Thanks.
> Add throughput controller for flush
> ---
>
> Key: HBASE-14969
> URL: https://issues.apache.org/jira/browse/HBASE-14969
> Project: HBase
> Issue Type: Improvement
>Affects Versions: 2.0.0
>Reporter: Yu Li
>Assignee: Yu Li
> Attachments: HBASE-14969.patch
>
>
> In HBASE-8329 we added a throughput controller for compaction, to avoid spike
> caused by huge IO pressure like network/disk overflow. However, even with
> this control on, we are still observing disk utils near 100%, and by analysis
> we think this is caused by flush, especially when we increase the setting of
> {{hbase.hstore.flusher.count}}
> In this JIRA, we propose to add throughput control feature for flush, as a
> supplement of HBASE-8329 to better control IO pressure.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
