[jira] [Commented] (HBASE-16679) Flush throughput controller: Minor perf change and fix flaky TestFlushWithThroughputController
[ https://issues.apache.org/jira/browse/HBASE-16679?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15521008#comment-15521008 ] Yu Li commented on HBASE-16679: --- Just noticed this one, nice catch and thanks for the patch [~appy] > Flush throughput controller: Minor perf change and fix flaky > TestFlushWithThroughputController > -- > > Key: HBASE-16679 > URL: https://issues.apache.org/jira/browse/HBASE-16679 > Project: HBase > Issue Type: Bug >Reporter: Appy >Assignee: Appy > Fix For: 2.0.0, 1.3.0, 1.4.0 > > Attachments: HBASE-16679.master.001.patch, > HBASE-16679.master.002.patch, HBASE-16679.master.003.patch > > > Minor perf change: > Calculate maxThroughputPerOperation outside of control() since start()() > are called only once per operation, but control can be called > hundreds/thousands of time. > Flaky test: > Problems in current test: > - writes only 2.5MB each iteration but control triggers sleep only every 1Mb > write (decided by HBASE_HSTORE_FLUSH_THROUGHPUT_CONTROL_CHECK_INTERVAL). > Either increase data written in each batch or decreasing this threshold for > better throughput control. > - We shouldn't be timing table disable/delete/create and populating data in > throughput calculations. > See the differences below. > With patch (total data written 30M) > run 1: > Throughput is: 1.0113841089709052 MB/s > Throughput w/o limit is: 14.665069580078125 MB/s > With 1M/s limit, flush use 29683ms; without limit, flush use 2130ms > run 2: > Throughput is: 1.0113841089709052 MB/s > Throughput w/o limit is: 14.665069580078125 MB/s > With 1M/s limit, flush use 29674ms; without limit, flush use 2027ms > Without patch (total data written 25M) > run 1: > Throughput is: 0.921681903523776 MB/s > Throughput w/o limit is: 4.06833346870301 MB/s > With 1M/s limit, flush use 27189ms; without limit, flush use 6159ms > run 2: > Throughput is: 0.9422982728478803 MB/s > Throughput w/o limit is: 4.047858424942981 MB/s > With 1M/s limit, flush use 26594ms; without limit, flush use 6190ms -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16679) Flush throughput controller: Minor perf change and fix flaky TestFlushWithThroughputController
[ https://issues.apache.org/jira/browse/HBASE-16679?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15518209#comment-15518209 ] Duo Zhang commented on HBASE-16679: --- Fine. :) > Flush throughput controller: Minor perf change and fix flaky > TestFlushWithThroughputController > -- > > Key: HBASE-16679 > URL: https://issues.apache.org/jira/browse/HBASE-16679 > Project: HBase > Issue Type: Bug >Reporter: Appy >Assignee: Appy > Fix For: 2.0.0, 1.3.0, 1.4.0 > > Attachments: HBASE-16679.master.001.patch, > HBASE-16679.master.002.patch, HBASE-16679.master.003.patch > > > Minor perf change: > Calculate maxThroughputPerOperation outside of control() since start()() > are called only once per operation, but control can be called > hundreds/thousands of time. > Flaky test: > Problems in current test: > - writes only 2.5MB each iteration but control triggers sleep only every 1Mb > write (decided by HBASE_HSTORE_FLUSH_THROUGHPUT_CONTROL_CHECK_INTERVAL). > Either increase data written in each batch or decreasing this threshold for > better throughput control. > - We shouldn't be timing table disable/delete/create and populating data in > throughput calculations. > See the differences below. > With patch (total data written 30M) > run 1: > Throughput is: 1.0113841089709052 MB/s > Throughput w/o limit is: 14.665069580078125 MB/s > With 1M/s limit, flush use 29683ms; without limit, flush use 2130ms > run 2: > Throughput is: 1.0113841089709052 MB/s > Throughput w/o limit is: 14.665069580078125 MB/s > With 1M/s limit, flush use 29674ms; without limit, flush use 2027ms > Without patch (total data written 25M) > run 1: > Throughput is: 0.921681903523776 MB/s > Throughput w/o limit is: 4.06833346870301 MB/s > With 1M/s limit, flush use 27189ms; without limit, flush use 6159ms > run 2: > Throughput is: 0.9422982728478803 MB/s > Throughput w/o limit is: 4.047858424942981 MB/s > With 1M/s limit, flush use 26594ms; without limit, flush use 6190ms -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16679) Flush throughput controller: Minor perf change and fix flaky TestFlushWithThroughputController
[ https://issues.apache.org/jira/browse/HBASE-16679?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15518078#comment-15518078 ] Hudson commented on HBASE-16679: FAILURE: Integrated in Jenkins build HBase-Trunk_matrix #1661 (See [https://builds.apache.org/job/HBase-Trunk_matrix/1661/]) HBASE-16679 Flush throughput controller: Minor perf change and fix flaky (appy: rev 408242430510c1cedb5ce7971f3ebb9f439870fe) * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/throttle/PressureAwareThroughputController.java * (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/throttle/TestFlushWithThroughputController.java > Flush throughput controller: Minor perf change and fix flaky > TestFlushWithThroughputController > -- > > Key: HBASE-16679 > URL: https://issues.apache.org/jira/browse/HBASE-16679 > Project: HBase > Issue Type: Bug >Reporter: Appy >Assignee: Appy > Fix For: 2.0.0, 1.3.0, 1.4.0 > > Attachments: HBASE-16679.master.001.patch, > HBASE-16679.master.002.patch, HBASE-16679.master.003.patch > > > Minor perf change: > Calculate maxThroughputPerOperation outside of control() since start()() > are called only once per operation, but control can be called > hundreds/thousands of time. > Flaky test: > Problems in current test: > - writes only 2.5MB each iteration but control triggers sleep only every 1Mb > write (decided by HBASE_HSTORE_FLUSH_THROUGHPUT_CONTROL_CHECK_INTERVAL). > Either increase data written in each batch or decreasing this threshold for > better throughput control. > - We shouldn't be timing table disable/delete/create and populating data in > throughput calculations. > See the differences below. > With patch (total data written 30M) > run 1: > Throughput is: 1.0113841089709052 MB/s > Throughput w/o limit is: 14.665069580078125 MB/s > With 1M/s limit, flush use 29683ms; without limit, flush use 2130ms > run 2: > Throughput is: 1.0113841089709052 MB/s > Throughput w/o limit is: 14.665069580078125 MB/s > With 1M/s limit, flush use 29674ms; without limit, flush use 2027ms > Without patch (total data written 25M) > run 1: > Throughput is: 0.921681903523776 MB/s > Throughput w/o limit is: 4.06833346870301 MB/s > With 1M/s limit, flush use 27189ms; without limit, flush use 6159ms > run 2: > Throughput is: 0.9422982728478803 MB/s > Throughput w/o limit is: 4.047858424942981 MB/s > With 1M/s limit, flush use 26594ms; without limit, flush use 6190ms -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16679) Flush throughput controller: Minor perf change and fix flaky TestFlushWithThroughputController
[ https://issues.apache.org/jira/browse/HBASE-16679?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15517973#comment-15517973 ] Hudson commented on HBASE-16679: FAILURE: Integrated in Jenkins build HBase-1.3-JDK8 #22 (See [https://builds.apache.org/job/HBase-1.3-JDK8/22/]) HBASE-16679 Flush throughput controller: Minor perf change and fix flaky (appy: rev b4f7ad62ef333e8a358cfad69f14205878591f57) * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/throttle/PressureAwareThroughputController.java * (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/throttle/TestFlushWithThroughputController.java > Flush throughput controller: Minor perf change and fix flaky > TestFlushWithThroughputController > -- > > Key: HBASE-16679 > URL: https://issues.apache.org/jira/browse/HBASE-16679 > Project: HBase > Issue Type: Bug >Reporter: Appy >Assignee: Appy > Fix For: 2.0.0, 1.3.0, 1.4.0 > > Attachments: HBASE-16679.master.001.patch, > HBASE-16679.master.002.patch, HBASE-16679.master.003.patch > > > Minor perf change: > Calculate maxThroughputPerOperation outside of control() since start()() > are called only once per operation, but control can be called > hundreds/thousands of time. > Flaky test: > Problems in current test: > - writes only 2.5MB each iteration but control triggers sleep only every 1Mb > write (decided by HBASE_HSTORE_FLUSH_THROUGHPUT_CONTROL_CHECK_INTERVAL). > Either increase data written in each batch or decreasing this threshold for > better throughput control. > - We shouldn't be timing table disable/delete/create and populating data in > throughput calculations. > See the differences below. > With patch (total data written 30M) > run 1: > Throughput is: 1.0113841089709052 MB/s > Throughput w/o limit is: 14.665069580078125 MB/s > With 1M/s limit, flush use 29683ms; without limit, flush use 2130ms > run 2: > Throughput is: 1.0113841089709052 MB/s > Throughput w/o limit is: 14.665069580078125 MB/s > With 1M/s limit, flush use 29674ms; without limit, flush use 2027ms > Without patch (total data written 25M) > run 1: > Throughput is: 0.921681903523776 MB/s > Throughput w/o limit is: 4.06833346870301 MB/s > With 1M/s limit, flush use 27189ms; without limit, flush use 6159ms > run 2: > Throughput is: 0.9422982728478803 MB/s > Throughput w/o limit is: 4.047858424942981 MB/s > With 1M/s limit, flush use 26594ms; without limit, flush use 6190ms -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16679) Flush throughput controller: Minor perf change and fix flaky TestFlushWithThroughputController
[ https://issues.apache.org/jira/browse/HBASE-16679?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15517739#comment-15517739 ] Hudson commented on HBASE-16679: FAILURE: Integrated in Jenkins build HBase-1.3-JDK7 #21 (See [https://builds.apache.org/job/HBase-1.3-JDK7/21/]) HBASE-16679 Flush throughput controller: Minor perf change and fix flaky (appy: rev b4f7ad62ef333e8a358cfad69f14205878591f57) * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/throttle/PressureAwareThroughputController.java * (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/throttle/TestFlushWithThroughputController.java > Flush throughput controller: Minor perf change and fix flaky > TestFlushWithThroughputController > -- > > Key: HBASE-16679 > URL: https://issues.apache.org/jira/browse/HBASE-16679 > Project: HBase > Issue Type: Bug >Reporter: Appy >Assignee: Appy > Fix For: 2.0.0, 1.3.0, 1.4.0 > > Attachments: HBASE-16679.master.001.patch, > HBASE-16679.master.002.patch, HBASE-16679.master.003.patch > > > Minor perf change: > Calculate maxThroughputPerOperation outside of control() since start()() > are called only once per operation, but control can be called > hundreds/thousands of time. > Flaky test: > Problems in current test: > - writes only 2.5MB each iteration but control triggers sleep only every 1Mb > write (decided by HBASE_HSTORE_FLUSH_THROUGHPUT_CONTROL_CHECK_INTERVAL). > Either increase data written in each batch or decreasing this threshold for > better throughput control. > - We shouldn't be timing table disable/delete/create and populating data in > throughput calculations. > See the differences below. > With patch (total data written 30M) > run 1: > Throughput is: 1.0113841089709052 MB/s > Throughput w/o limit is: 14.665069580078125 MB/s > With 1M/s limit, flush use 29683ms; without limit, flush use 2130ms > run 2: > Throughput is: 1.0113841089709052 MB/s > Throughput w/o limit is: 14.665069580078125 MB/s > With 1M/s limit, flush use 29674ms; without limit, flush use 2027ms > Without patch (total data written 25M) > run 1: > Throughput is: 0.921681903523776 MB/s > Throughput w/o limit is: 4.06833346870301 MB/s > With 1M/s limit, flush use 27189ms; without limit, flush use 6159ms > run 2: > Throughput is: 0.9422982728478803 MB/s > Throughput w/o limit is: 4.047858424942981 MB/s > With 1M/s limit, flush use 26594ms; without limit, flush use 6190ms -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16679) Flush throughput controller: Minor perf change and fix flaky TestFlushWithThroughputController
[
https://issues.apache.org/jira/browse/HBASE-16679?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15517194#comment-15517194
]
Appy commented on HBASE-16679:
--
Committing.
Ran above failing/timed out tests locally.
TestBlockEvictionFromClient is flaky while other tests timed out probably
becuase of java.lang.RuntimeException: java.lang.OutOfMemoryError.
The changes in this patch doesn't affect other tests since these pressure aware
controllers are not enabled by default.
{noformat}
---
T E S T S
---
Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=256m;
support was removed in 8.0
Running org.apache.hadoop.hbase.namespace.TestNamespaceAuditor
Tests run: 12, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 141.852 sec -
in org.apache.hadoop.hbase.namespace.TestNamespaceAuditor
Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=256m;
support was removed in 8.0
Running
org.apache.hadoop.hbase.security.visibility.TestVisibilityLabelsWithDeletes
Tests run: 44, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 91.247 sec -
in org.apache.hadoop.hbase.security.visibility.TestVisibilityLabelsWithDeletes
Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=256m;
support was removed in 8.0
Running org.apache.hadoop.hbase.wal.TestBoundedRegionGroupingStrategy
Tests run: 10, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 38.128 sec -
in org.apache.hadoop.hbase.wal.TestBoundedRegionGroupingStrategy
Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=256m;
support was removed in 8.0
Running org.apache.hadoop.hbase.wal.TestWALSplit
Tests run: 32, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 59.011 sec -
in org.apache.hadoop.hbase.wal.TestWALSplit
Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=256m;
support was removed in 8.0
Running org.apache.hadoop.hbase.wal.TestWALSplitCompressed
Tests run: 32, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 57.131 sec -
in org.apache.hadoop.hbase.wal.TestWALSplitCompressed
Results :
Tests run: 130, Failures: 0, Errors: 0, Skipped: 0
{noformat}
> Flush throughput controller: Minor perf change and fix flaky
> TestFlushWithThroughputController
> --
>
> Key: HBASE-16679
> URL: https://issues.apache.org/jira/browse/HBASE-16679
> Project: HBase
> Issue Type: Bug
>Reporter: Appy
>Assignee: Appy
> Attachments: HBASE-16679.master.001.patch,
> HBASE-16679.master.002.patch, HBASE-16679.master.003.patch
>
>
> Minor perf change:
> Calculate maxThroughputPerOperation outside of control() since start()()
> are called only once per operation, but control can be called
> hundreds/thousands of time.
> Flaky test:
> Problems in current test:
> - writes only 2.5MB each iteration but control triggers sleep only every 1Mb
> write (decided by HBASE_HSTORE_FLUSH_THROUGHPUT_CONTROL_CHECK_INTERVAL).
> Either increase data written in each batch or decreasing this threshold for
> better throughput control.
> - We shouldn't be timing table disable/delete/create and populating data in
> throughput calculations.
> See the differences below.
> With patch (total data written 30M)
> run 1:
> Throughput is: 1.0113841089709052 MB/s
> Throughput w/o limit is: 14.665069580078125 MB/s
> With 1M/s limit, flush use 29683ms; without limit, flush use 2130ms
> run 2:
> Throughput is: 1.0113841089709052 MB/s
> Throughput w/o limit is: 14.665069580078125 MB/s
> With 1M/s limit, flush use 29674ms; without limit, flush use 2027ms
> Without patch (total data written 25M)
> run 1:
> Throughput is: 0.921681903523776 MB/s
> Throughput w/o limit is: 4.06833346870301 MB/s
> With 1M/s limit, flush use 27189ms; without limit, flush use 6159ms
> run 2:
> Throughput is: 0.9422982728478803 MB/s
> Throughput w/o limit is: 4.047858424942981 MB/s
> With 1M/s limit, flush use 26594ms; without limit, flush use 6190ms
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-16679) Flush throughput controller: Minor perf change and fix flaky TestFlushWithThroughputController
[ https://issues.apache.org/jira/browse/HBASE-16679?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15517143#comment-15517143 ] Appy commented on HBASE-16679: -- I see. Thanks for explaining. > Flush throughput controller: Minor perf change and fix flaky > TestFlushWithThroughputController > -- > > Key: HBASE-16679 > URL: https://issues.apache.org/jira/browse/HBASE-16679 > Project: HBase > Issue Type: Bug >Reporter: Appy >Assignee: Appy > Attachments: HBASE-16679.master.001.patch, > HBASE-16679.master.002.patch, HBASE-16679.master.003.patch > > > Minor perf change: > Calculate maxThroughputPerOperation outside of control() since start()() > are called only once per operation, but control can be called > hundreds/thousands of time. > Flaky test: > Problems in current test: > - writes only 2.5MB each iteration but control triggers sleep only every 1Mb > write (decided by HBASE_HSTORE_FLUSH_THROUGHPUT_CONTROL_CHECK_INTERVAL). > Either increase data written in each batch or decreasing this threshold for > better throughput control. > - We shouldn't be timing table disable/delete/create and populating data in > throughput calculations. > See the differences below. > With patch (total data written 30M) > run 1: > Throughput is: 1.0113841089709052 MB/s > Throughput w/o limit is: 14.665069580078125 MB/s > With 1M/s limit, flush use 29683ms; without limit, flush use 2130ms > run 2: > Throughput is: 1.0113841089709052 MB/s > Throughput w/o limit is: 14.665069580078125 MB/s > With 1M/s limit, flush use 29674ms; without limit, flush use 2027ms > Without patch (total data written 25M) > run 1: > Throughput is: 0.921681903523776 MB/s > Throughput w/o limit is: 4.06833346870301 MB/s > With 1M/s limit, flush use 27189ms; without limit, flush use 6159ms > run 2: > Throughput is: 0.9422982728478803 MB/s > Throughput w/o limit is: 4.047858424942981 MB/s > With 1M/s limit, flush use 26594ms; without limit, flush use 6190ms -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16679) Flush throughput controller: Minor perf change and fix flaky TestFlushWithThroughputController
[ https://issues.apache.org/jira/browse/HBASE-16679?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15516788#comment-15516788 ] stack commented on HBASE-16679: --- Excellent [~appy] +1 Lets try it. This is the flakey that fails most going by your flakey report (available to all off the home page for trunk builds up on jenkins) [~Apache9], suggest we let @appy commit his own s*** -- smile. > Flush throughput controller: Minor perf change and fix flaky > TestFlushWithThroughputController > -- > > Key: HBASE-16679 > URL: https://issues.apache.org/jira/browse/HBASE-16679 > Project: HBase > Issue Type: Bug >Reporter: Appy >Assignee: Appy > Attachments: HBASE-16679.master.001.patch, > HBASE-16679.master.002.patch, HBASE-16679.master.003.patch > > > Minor perf change: > Calculate maxThroughputPerOperation outside of control() since start()() > are called only once per operation, but control can be called > hundreds/thousands of time. > Flaky test: > Problems in current test: > - writes only 2.5MB each iteration but control triggers sleep only every 1Mb > write (decided by HBASE_HSTORE_FLUSH_THROUGHPUT_CONTROL_CHECK_INTERVAL). > Either increase data written in each batch or decreasing this threshold for > better throughput control. > - We shouldn't be timing table disable/delete/create and populating data in > throughput calculations. > See the differences below. > With patch (total data written 30M) > run 1: > Throughput is: 1.0113841089709052 MB/s > Throughput w/o limit is: 14.665069580078125 MB/s > With 1M/s limit, flush use 29683ms; without limit, flush use 2130ms > run 2: > Throughput is: 1.0113841089709052 MB/s > Throughput w/o limit is: 14.665069580078125 MB/s > With 1M/s limit, flush use 29674ms; without limit, flush use 2027ms > Without patch (total data written 25M) > run 1: > Throughput is: 0.921681903523776 MB/s > Throughput w/o limit is: 4.06833346870301 MB/s > With 1M/s limit, flush use 27189ms; without limit, flush use 6159ms > run 2: > Throughput is: 0.9422982728478803 MB/s > Throughput w/o limit is: 4.047858424942981 MB/s > With 1M/s limit, flush use 26594ms; without limit, flush use 6190ms -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16679) Flush throughput controller: Minor perf change and fix flaky TestFlushWithThroughputController
[
https://issues.apache.org/jira/browse/HBASE-16679?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15516311#comment-15516311
]
Hadoop QA commented on HBASE-16679:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 27s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
14s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 42s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
47s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
14s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
51s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 31s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
55s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 42s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 42s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
45s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
14s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
34m 34s {color} | {color:green} Patch does not cause any errors with Hadoop
2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.1 2.6.2 2.6.3 2.7.1. {color} |
| {color:green}+1{color} | {color:green} hbaseprotoc {color} | {color:green} 0m
16s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
12s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 31s
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 109m 14s
{color} | {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
17s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 158m 51s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hbase.client.TestBlockEvictionFromClient |
| Timed out junit tests |
org.apache.hadoop.hbase.namespace.TestNamespaceAuditor |
| | org.apache.hadoop.hbase.wal.TestWALSplitCompressed |
| |
org.apache.hadoop.hbase.security.visibility.TestVisibilityLabelsWithDeletes |
| | org.apache.hadoop.hbase.wal.TestBoundedRegionGroupingStrategy |
| | org.apache.hadoop.hbase.wal.TestWALSplit |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.11.2 Server=1.11.2 Image:yetus/hbase:7bda515 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12830015/HBASE-16679.master.003.patch
|
| JIRA Issue | HBASE-16679 |
| Optional Tests | asflicense javac javadoc unit findbugs hadoopcheck
hbaseanti checkstyle compile |
| uname | Linux e7e3e65fd48e 3.13.0-92-generic #139-Ubuntu SMP Tue Jun 28
20:42:26 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
|
| git revision | master / 946c1ed |
| Default Java | 1.8.0_101 |
| findbugs | v3.0.0 |
| unit |
https://builds.apache.org/job/PreCommit-HBASE-Build/3680/artifact/patchprocess/patch-unit-hbase-server.txt
|
| unit test logs |
https://builds.apache.org/job/PreCommit-HBASE-Build/3680/artifact/patchprocess/patch-unit-hbase-server.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HBASE-Build/3680/testReport/ |
| modules | C: hbase-server U: hbase-server |
| Console output
[jira] [Commented] (HBASE-16679) Flush throughput controller: Minor perf change and fix flaky TestFlushWithThroughputController
[ https://issues.apache.org/jira/browse/HBASE-16679?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15515847#comment-15515847 ] Duo Zhang commented on HBASE-16679: --- +1 on v3. Ping [~stack]. I can help committing if you do not have time. Thanks. > Flush throughput controller: Minor perf change and fix flaky > TestFlushWithThroughputController > -- > > Key: HBASE-16679 > URL: https://issues.apache.org/jira/browse/HBASE-16679 > Project: HBase > Issue Type: Bug >Reporter: Appy >Assignee: Appy > Attachments: HBASE-16679.master.001.patch, > HBASE-16679.master.002.patch, HBASE-16679.master.003.patch > > > Minor perf change: > Calculate maxThroughputPerOperation outside of control() since start()() > are called only once per operation, but control can be called > hundreds/thousands of time. > Flaky test: > Problems in current test: > - writes only 2.5MB each iteration but control triggers sleep only every 1Mb > write (decided by HBASE_HSTORE_FLUSH_THROUGHPUT_CONTROL_CHECK_INTERVAL). > Either increase data written in each batch or decreasing this threshold for > better throughput control. > - We shouldn't be timing table disable/delete/create and populating data in > throughput calculations. > See the differences below. > With patch (total data written 30M) > run 1: > Throughput is: 1.0113841089709052 MB/s > Throughput w/o limit is: 14.665069580078125 MB/s > With 1M/s limit, flush use 29683ms; without limit, flush use 2130ms > run 2: > Throughput is: 1.0113841089709052 MB/s > Throughput w/o limit is: 14.665069580078125 MB/s > With 1M/s limit, flush use 29674ms; without limit, flush use 2027ms > Without patch (total data written 25M) > run 1: > Throughput is: 0.921681903523776 MB/s > Throughput w/o limit is: 4.06833346870301 MB/s > With 1M/s limit, flush use 27189ms; without limit, flush use 6159ms > run 2: > Throughput is: 0.9422982728478803 MB/s > Throughput w/o limit is: 4.047858424942981 MB/s > With 1M/s limit, flush use 26594ms; without limit, flush use 6190ms -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16679) Flush throughput controller: Minor perf change and fix flaky TestFlushWithThroughputController
[ https://issues.apache.org/jira/browse/HBASE-16679?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15515816#comment-15515816 ] Duo Zhang commented on HBASE-16679: --- Here in this class ,and for now, it is not a problem. But it is a bad practice that update a static field from a non-static method. You will see a findbugs warning if you do that :) > Flush throughput controller: Minor perf change and fix flaky > TestFlushWithThroughputController > -- > > Key: HBASE-16679 > URL: https://issues.apache.org/jira/browse/HBASE-16679 > Project: HBase > Issue Type: Bug >Reporter: Appy >Assignee: Appy > Attachments: HBASE-16679.master.001.patch, > HBASE-16679.master.002.patch > > > Minor perf change: > Calculate maxThroughputPerOperation outside of control() since start()() > are called only once per operation, but control can be called > hundreds/thousands of time. > Flaky test: > Problems in current test: > - writes only 2.5MB each iteration but control triggers sleep only every 1Mb > write (decided by HBASE_HSTORE_FLUSH_THROUGHPUT_CONTROL_CHECK_INTERVAL). > Either increase data written in each batch or decreasing this threshold for > better throughput control. > - We shouldn't be timing table disable/delete/create and populating data in > throughput calculations. > See the differences below. > With patch (total data written 30M) > run 1: > Throughput is: 1.0113841089709052 MB/s > Throughput w/o limit is: 14.665069580078125 MB/s > With 1M/s limit, flush use 29683ms; without limit, flush use 2130ms > run 2: > Throughput is: 1.0113841089709052 MB/s > Throughput w/o limit is: 14.665069580078125 MB/s > With 1M/s limit, flush use 29674ms; without limit, flush use 2027ms > Without patch (total data written 25M) > run 1: > Throughput is: 0.921681903523776 MB/s > Throughput w/o limit is: 4.06833346870301 MB/s > With 1M/s limit, flush use 27189ms; without limit, flush use 6159ms > run 2: > Throughput is: 0.9422982728478803 MB/s > Throughput w/o limit is: 4.047858424942981 MB/s > With 1M/s limit, flush use 26594ms; without limit, flush use 6190ms -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16679) Flush throughput controller: Minor perf change and fix flaky TestFlushWithThroughputController
[ https://issues.apache.org/jira/browse/HBASE-16679?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15515802#comment-15515802 ] Appy commented on HBASE-16679: -- Done. We are anyways restarting clusters, stack had the good suggestion to not muck around with same configuration as it can cause silly bugs. So resetting it every time. And static or not, it doesn't really matter, right? (but i have made it non-static). > Flush throughput controller: Minor perf change and fix flaky > TestFlushWithThroughputController > -- > > Key: HBASE-16679 > URL: https://issues.apache.org/jira/browse/HBASE-16679 > Project: HBase > Issue Type: Bug >Reporter: Appy >Assignee: Appy > Attachments: HBASE-16679.master.001.patch, > HBASE-16679.master.002.patch > > > Minor perf change: > Calculate maxThroughputPerOperation outside of control() since start()() > are called only once per operation, but control can be called > hundreds/thousands of time. > Flaky test: > Problems in current test: > - writes only 2.5MB each iteration but control triggers sleep only every 1Mb > write (decided by HBASE_HSTORE_FLUSH_THROUGHPUT_CONTROL_CHECK_INTERVAL). > Either increase data written in each batch or decreasing this threshold for > better throughput control. > - We shouldn't be timing table disable/delete/create and populating data in > throughput calculations. > See the differences below. > With patch (total data written 30M) > run 1: > Throughput is: 1.0113841089709052 MB/s > Throughput w/o limit is: 14.665069580078125 MB/s > With 1M/s limit, flush use 29683ms; without limit, flush use 2130ms > run 2: > Throughput is: 1.0113841089709052 MB/s > Throughput w/o limit is: 14.665069580078125 MB/s > With 1M/s limit, flush use 29674ms; without limit, flush use 2027ms > Without patch (total data written 25M) > run 1: > Throughput is: 0.921681903523776 MB/s > Throughput w/o limit is: 4.06833346870301 MB/s > With 1M/s limit, flush use 27189ms; without limit, flush use 6159ms > run 2: > Throughput is: 0.9422982728478803 MB/s > Throughput w/o limit is: 4.047858424942981 MB/s > With 1M/s limit, flush use 26594ms; without limit, flush use 6190ms -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16679) Flush throughput controller: Minor perf change and fix flaky TestFlushWithThroughputController
[
https://issues.apache.org/jira/browse/HBASE-16679?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15512538#comment-15512538
]
Hadoop QA commented on HBASE-16679:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 15s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
42s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 38s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
48s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
17s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
55s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 30s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
49s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 38s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 38s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
48s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
14s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
28m 41s {color} | {color:green} Patch does not cause any errors with Hadoop
2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.1 2.6.2 2.6.3 2.7.1. {color} |
| {color:green}+1{color} | {color:green} hbaseprotoc {color} | {color:green} 0m
14s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m 2s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 27s
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 102m 40s
{color} | {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
18s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 145m 28s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hbase.regionserver.TestHRegionWithInMemoryFlush |
| Timed out junit tests |
org.apache.hadoop.hbase.snapshot.TestSnapshotClientRetries |
| | org.apache.hadoop.hbase.snapshot.TestMobSecureExportSnapshot |
| | org.apache.hadoop.hbase.snapshot.TestRestoreFlushSnapshotFromClient |
| | org.apache.hadoop.hbase.client.TestHCM |
| | org.apache.hadoop.hbase.snapshot.TestMobFlushSnapshotFromClient |
| | org.apache.hadoop.hbase.snapshot.TestFlushSnapshotFromClient |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.11.2 Server=1.11.2 Image:yetus/hbase:7bda515 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12829752/HBASE-16679.master.002.patch
|
| JIRA Issue | HBASE-16679 |
| Optional Tests | asflicense javac javadoc unit findbugs hadoopcheck
hbaseanti checkstyle compile |
| uname | Linux 19a654900c21 3.13.0-92-generic #139-Ubuntu SMP Tue Jun 28
20:42:26 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build@2/component/dev-support/hbase-personality.sh
|
| git revision | master / c9d0d11 |
| Default Java | 1.8.0_101 |
| findbugs | v3.0.0 |
| unit |
https://builds.apache.org/job/PreCommit-HBASE-Build/3649/artifact/patchprocess/patch-unit-hbase-server.txt
|
| unit test logs |
https://builds.apache.org/job/PreCommit-HBASE-Build/3649/artifact/patchprocess/patch-unit-hbase-server.txt
|
| Test Results |
[jira] [Commented] (HBASE-16679) Flush throughput controller: Minor perf change and fix flaky TestFlushWithThroughputController
[
https://issues.apache.org/jira/browse/HBASE-16679?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15512174#comment-15512174
]
Duo Zhang commented on HBASE-16679:
---
The {{maxThroughputPerOperation}} should be volatile.
And the {{HBaseTestingUtility}} does not need to be static as you recreate it
in setUp?
Thanks.
> Flush throughput controller: Minor perf change and fix flaky
> TestFlushWithThroughputController
> --
>
> Key: HBASE-16679
> URL: https://issues.apache.org/jira/browse/HBASE-16679
> Project: HBase
> Issue Type: Bug
>Reporter: Appy
>Assignee: Appy
> Attachments: HBASE-16679.master.001.patch,
> HBASE-16679.master.002.patch
>
>
> Minor perf change:
> Calculate maxThroughputPerOperation outside of control() since start()()
> are called only once per operation, but control can be called
> hundreds/thousands of time.
> Flaky test:
> Problems in current test:
> - writes only 2.5MB each iteration but control triggers sleep only every 1Mb
> write (decided by HBASE_HSTORE_FLUSH_THROUGHPUT_CONTROL_CHECK_INTERVAL).
> Either increase data written in each batch or decreasing this threshold for
> better throughput control.
> - We shouldn't be timing table disable/delete/create and populating data in
> throughput calculations.
> See the differences below.
> With patch (total data written 30M)
> run 1:
> Throughput is: 1.0113841089709052 MB/s
> Throughput w/o limit is: 14.665069580078125 MB/s
> With 1M/s limit, flush use 29683ms; without limit, flush use 2130ms
> run 2:
> Throughput is: 1.0113841089709052 MB/s
> Throughput w/o limit is: 14.665069580078125 MB/s
> With 1M/s limit, flush use 29674ms; without limit, flush use 2027ms
> Without patch (total data written 25M)
> run 1:
> Throughput is: 0.921681903523776 MB/s
> Throughput w/o limit is: 4.06833346870301 MB/s
> With 1M/s limit, flush use 27189ms; without limit, flush use 6159ms
> run 2:
> Throughput is: 0.9422982728478803 MB/s
> Throughput w/o limit is: 4.047858424942981 MB/s
> With 1M/s limit, flush use 26594ms; without limit, flush use 6190ms
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
