[jira] [Commented] (HBASE-15222) Use HDR histograms rather than hadoop or yammer's
[
https://issues.apache.org/jira/browse/HBASE-15222?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15154084#comment-15154084
]
Hadoop QA commented on HBASE-15222:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 0s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 1s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m 0s
{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
53s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 3m 21s
{color} | {color:green} master passed with JDK v1.8.0_72 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 3m 18s
{color} | {color:green} master passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 16m
47s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 1m
46s {color} | {color:green} master passed {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 8m 6s
{color} | {color:red} branch/. no findbugs output file
(./target/findbugsXml.xml) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 3m 21s
{color} | {color:green} master passed with JDK v1.8.0_72 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 3m 31s
{color} | {color:green} master passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
5s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 3m 31s
{color} | {color:green} the patch passed with JDK v1.8.0_72 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 3m 31s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 3m 20s
{color} | {color:green} the patch passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 3m 20s
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 11m 30s
{color} | {color:red} Patch generated 8 new checkstyle issues in root (total
was 244, now 230). {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 1m 3s
{color} | {color:red} Patch generated 1 new checkstyle issues in hbase-common
(total was 1, now 2). {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m 13s
{color} | {color:red} Patch generated 3 new checkstyle issues in
hbase-hadoop2-compat (total was 48, now 32). {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 4m 0s
{color} | {color:red} Patch generated 4 new checkstyle issues in hbase-server
(total was 194, now 195). {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 1m
48s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m 1s
{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
22m 52s {color} | {color:green} Patch does not cause any errors with Hadoop
2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.1 2.6.2 2.6.3 2.7.1. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 8m 1s
{color} | {color:red} patch/. no findbugs output file
(./target/findbugsXml.xml) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 3m 6s
{color} | {color:green} the patch passed with JDK v1.8.0_72 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 3m 32s
{color} | {color:green} the patch passed with JDK v1.7.0_95 {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 97m 4s {color}
| {color:red} root in the patch failed with JDK v1.8.0_72. {color} |
| {color:green}+1{color} |
[jira] [Commented] (HBASE-15222) Use HDR histograms rather than hadoop or yammer's
[
https://issues.apache.org/jira/browse/HBASE-15222?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15153751#comment-15153751
]
Hadoop QA commented on HBASE-15222:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 1s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m 0s
{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
20s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 3m 43s
{color} | {color:green} master passed with JDK v1.8.0_72 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 3m 47s
{color} | {color:green} master passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 18m
41s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 2m
0s {color} | {color:green} master passed {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 9m 5s
{color} | {color:red} branch/. no findbugs output file
(./target/findbugsXml.xml) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 3m 19s
{color} | {color:green} master passed with JDK v1.8.0_72 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 3m 56s
{color} | {color:green} master passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
32s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 3m 40s
{color} | {color:green} the patch passed with JDK v1.8.0_72 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 3m 40s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 3m 50s
{color} | {color:green} the patch passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 3m 50s
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 12m 38s
{color} | {color:red} Patch generated 8 new checkstyle issues in root (total
was 244, now 230). {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 1m 7s
{color} | {color:red} Patch generated 1 new checkstyle issues in hbase-common
(total was 1, now 2). {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m 15s
{color} | {color:red} Patch generated 3 new checkstyle issues in
hbase-hadoop2-compat (total was 48, now 32). {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 4m 1s
{color} | {color:red} Patch generated 4 new checkstyle issues in hbase-server
(total was 194, now 195). {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 2m
29s {color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m 0s
{color} | {color:red} The patch has 1 line(s) that end in whitespace. Use git
apply --whitespace=fix. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m 2s
{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
30m 27s {color} | {color:green} Patch does not cause any errors with Hadoop
2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.1 2.6.2 2.6.3 2.7.1. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 10m 37s
{color} | {color:red} patch/. no findbugs output file
(./target/findbugsXml.xml) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 3m 57s
{color} | {color:green} the patch passed with JDK v1.8.0_72 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 4m 42s
{color} | {color:green} the patch passed with JDK v1.7.0_95 {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 90m 35s {color}
| {color:red} root in the patch failed with JDK v1.8.0_72.
[jira] [Commented] (HBASE-15222) Use HDR histograms rather than hadoop or yammer's
[ https://issues.apache.org/jira/browse/HBASE-15222?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15153036#comment-15153036 ] Elliott Clark commented on HBASE-15222: --- https://reviews.facebook.net/D54381 I added the transition to counter in a few places. I also transitioned block cache to HDR histograms. > Use HDR histograms rather than hadoop or yammer's > - > > Key: HBASE-15222 > URL: https://issues.apache.org/jira/browse/HBASE-15222 > Project: HBase > Issue Type: Bug > Components: metrics >Affects Versions: 2.0.0, 1.2.0, 1.3.0 >Reporter: Elliott Clark >Assignee: Elliott Clark >Priority: Critical > Fix For: 2.0.0, 1.3.0 > > Attachments: HBASE-15222.patch > > > Running the benchmarks now, but it looks like the results are pretty extreme. > The locking in our histograms is pretty extreme. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15222) Use HDR histograms rather than hadoop or yammer's
[ https://issues.apache.org/jira/browse/HBASE-15222?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15152918#comment-15152918 ] Mikhail Antonov commented on HBASE-15222: - Looks good to me. > Use HDR histograms rather than hadoop or yammer's > - > > Key: HBASE-15222 > URL: https://issues.apache.org/jira/browse/HBASE-15222 > Project: HBase > Issue Type: Bug > Components: metrics >Affects Versions: 2.0.0, 1.2.0, 1.3.0 >Reporter: Elliott Clark >Assignee: Elliott Clark >Priority: Critical > Fix For: 2.0.0, 1.3.0 > > Attachments: HBASE-15222.patch > > > Running the benchmarks now, but it looks like the results are pretty extreme. > The locking in our histograms is pretty extreme. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15222) Use HDR histograms rather than hadoop or yammer's
[ https://issues.apache.org/jira/browse/HBASE-15222?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15147948#comment-15147948 ] Mikhail Antonov commented on HBASE-15222: - Bumping priority to critical as I've seen on some of our clusters that when nun of active handlers goes above certain threshold (35 or 40 threads) user-cpu jumpt to 90+% and more than half (and sometimes 80%) of threads are running histogram updates :( > Use HDR histograms rather than hadoop or yammer's > - > > Key: HBASE-15222 > URL: https://issues.apache.org/jira/browse/HBASE-15222 > Project: HBase > Issue Type: Bug > Components: metrics >Affects Versions: 2.0.0, 1.2.0, 1.3.0 >Reporter: Elliott Clark >Assignee: Elliott Clark >Priority: Critical > Fix For: 2.0.0, 1.3.0 > > > Running the benchmarks now, but it looks like the results are pretty extreme. > The locking in our histograms is pretty extreme. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15222) Use HDR histograms rather than hadoop or yammer's
[ https://issues.apache.org/jira/browse/HBASE-15222?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15135086#comment-15135086 ] Sean Busbey commented on HBASE-15222: - probably not within scope for this ticket, but an interesting side effect of using hte HDRHistogram library is we could give folks the option of locally outputting its log format so they could do analysis on arbitrary time spans. > Use HDR histograms rather than hadoop or yammer's > - > > Key: HBASE-15222 > URL: https://issues.apache.org/jira/browse/HBASE-15222 > Project: HBase > Issue Type: Bug > Components: metrics >Affects Versions: 2.0.0, 1.2.0, 1.3.0 >Reporter: Elliott Clark >Assignee: Elliott Clark > Fix For: 2.0.0, 1.3.0 > > > Running the benchmarks now, but it looks like the results are pretty extreme. > The locking in our histograms is pretty extreme. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15222) Use HDR histograms rather than hadoop or yammer's
[
https://issues.apache.org/jira/browse/HBASE-15222?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15135084#comment-15135084
]
Elliott Clark commented on HBASE-15222:
---
Initial results:
{code}
Benchmark Mode Cnt Score Error Units
HistogramBenchmark.testHDRAtomic avgt21117.988 ns/op
HistogramBenchmark.testHDRConcurrent avgt22330.014 ns/op
HistogramBenchmark.testNormal avgt26085.187 ns/op
HistogramBenchmark.testQuant avgt2 397337.211 ns/op
{code}
So HDR is much faster. I'm going to re-run the test on server hardware with
more cores than my laptop. I'm also going to run it for longer.
> Use HDR histograms rather than hadoop or yammer's
> -
>
> Key: HBASE-15222
> URL: https://issues.apache.org/jira/browse/HBASE-15222
> Project: HBase
> Issue Type: Bug
> Components: metrics
>Affects Versions: 2.0.0, 1.2.0, 1.3.0
>Reporter: Elliott Clark
>Assignee: Elliott Clark
> Fix For: 2.0.0, 1.3.0
>
>
> Running the benchmarks now, but it looks like the results are pretty extreme.
> The locking in our histograms is pretty extreme.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-15222) Use HDR histograms rather than hadoop or yammer's
[
https://issues.apache.org/jira/browse/HBASE-15222?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15135040#comment-15135040
]
Elliott Clark commented on HBASE-15222:
---
We recently saw a lot of contention on a loaded server. That contention was
almost completely in the metrics subsystem.
{code}
Thread 80 (B.defaultRpcServer.handler=37,queue=11,port=16020):
State: RUNNABLE
Blocked count: 664893
Waited count: 10166003
Stack:
java.lang.ThreadLocal$ThreadLocalMap.expungeStaleEntry(ThreadLocal.java:617)
java.lang.ThreadLocal$ThreadLocalMap.remove(ThreadLocal.java:499)
java.lang.ThreadLocal$ThreadLocalMap.access$200(ThreadLocal.java:298)
java.lang.ThreadLocal.remove(ThreadLocal.java:222)
java.util.concurrent.locks.ReentrantReadWriteLock$Sync.tryReleaseShared(ReentrantReadWriteLock.java:426)
java.util.concurrent.locks.AbstractQueuedSynchronizer.releaseShared(AbstractQueuedSynchronizer.java:1341)
java.util.concurrent.locks.ReentrantReadWriteLock$ReadLock.unlock(ReentrantReadWriteLock.java:881)
com.yammer.metrics.stats.ExponentiallyDecayingSample.unlockForRegularUsage(ExponentiallyDecayingSample.java:196)
com.yammer.metrics.stats.ExponentiallyDecayingSample.update(ExponentiallyDecayingSample.java:113)
com.yammer.metrics.stats.ExponentiallyDecayingSample.update(ExponentiallyDecayingSample.java:81)
org.apache.hadoop.metrics2.lib.MutableHistogram.add(MutableHistogram.java:69)
org.apache.hadoop.metrics2.lib.MutableRangeHistogram.add(MutableRangeHistogram.java:58)
org.apache.hadoop.hbase.regionserver.MetricsRegionServerSourceImpl.updateGet(MetricsRegionServerSourceImpl.java:108)
org.apache.hadoop.hbase.regionserver.MetricsRegionServer.updateGet(MetricsRegionServer.java:79)
org.apache.hadoop.hbase.regionserver.RSRpcServices.doNonAtomicRegionMutation(RSRpcServices.java:628)
org.apache.hadoop.hbase.regionserver.RSRpcServices.multi(RSRpcServices.java:2138)
org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:33656)
org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2170)
org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:109)
org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:133)
Thread 79 (B.defaultRpcServer.handler=36,queue=10,port=16020):
State: RUNNABLE
Blocked count: 546694
Waited count: 8942562
Stack:
java.lang.ThreadLocal$ThreadLocalMap.expungeStaleEntry(ThreadLocal.java:617)
java.lang.ThreadLocal$ThreadLocalMap.remove(ThreadLocal.java:499)
java.lang.ThreadLocal$ThreadLocalMap.access$200(ThreadLocal.java:298)
java.lang.ThreadLocal.remove(ThreadLocal.java:222)
java.util.concurrent.locks.ReentrantReadWriteLock$Sync.tryReleaseShared(ReentrantReadWriteLock.java:426)
java.util.concurrent.locks.AbstractQueuedSynchronizer.releaseShared(AbstractQueuedSynchronizer.java:1341)
java.util.concurrent.locks.ReentrantReadWriteLock$ReadLock.unlock(ReentrantReadWriteLock.java:881)
com.yammer.metrics.stats.ExponentiallyDecayingSample.unlockForRegularUsage(ExponentiallyDecayingSample.java:196)
com.yammer.metrics.stats.ExponentiallyDecayingSample.update(ExponentiallyDecayingSample.java:113)
com.yammer.metrics.stats.ExponentiallyDecayingSample.update(ExponentiallyDecayingSample.java:81)
org.apache.hadoop.metrics2.lib.MutableHistogram.add(MutableHistogram.java:69)
org.apache.hadoop.metrics2.lib.MutableRangeHistogram.add(MutableRangeHistogram.java:58)
org.apache.hadoop.hbase.regionserver.MetricsRegionServerSourceImpl.updateGet(MetricsRegionServerSourceImpl.java:108)
org.apache.hadoop.hbase.regionserver.MetricsRegionServer.updateGet(MetricsRegionServer.java:79)
org.apache.hadoop.hbase.regionserver.RSRpcServices.doNonAtomicRegionMutation(RSRpcServices.java:628)
org.apache.hadoop.hbase.regionserver.RSRpcServices.multi(RSRpcServices.java:2138)
org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:33656)
org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2170)
org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:109)
org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:133)
Thread 78 (B.defaultRpcServer.handler=35,queue=9,port=16020):
State: RUNNABLE
Blocked count: 546871
Waited count: 8886334
Stack:
java.lang.ThreadLocal$ThreadLocalMap.expungeStaleEntry(ThreadLocal.java:617)
java.lang.ThreadLocal$ThreadLocalMap.remove(ThreadLocal.java:499)
java.lang.ThreadLocal$ThreadLocalMap.access$200(ThreadLocal.java:298)
java.lang.ThreadLocal.remove(ThreadLocal.java:222)
java.util.concurrent.locks.ReentrantReadWriteLock$Sync.tryReleaseShared(ReentrantReadWriteLock.java:426)
java.util.concurrent.locks.AbstractQueuedSynchronizer.releaseShared(AbstractQueuedSynchronizer.java:1341)
[jira] [Commented] (HBASE-15222) Use HDR histograms rather than hadoop or yammer's
[
https://issues.apache.org/jira/browse/HBASE-15222?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15135301#comment-15135301
]
Elliott Clark commented on HBASE-15222:
---
Longer benchmark run:
{code}
Benchmark Mode Cnt ScoreError Units
HistogramBenchmark.testHDRAtomic avgt 254226.755 ±771.162 ns/op
HistogramBenchmark.testHDRConcurrent avgt 256478.152 ±275.396 ns/op
HistogramBenchmark.testNormal avgt 25 15977.900 ± 1076.541 ns/op
HistogramBenchmark.testQuant avgt 25 223378.087 ± 215236.829 ns/op
{code}
So our Quantile code is s slow that it shouldn't be used.
Using HDR's versions of histograms at higher concurrency can half the overhead,
or better.
Let me check and see how just yammer's stuff works.
> Use HDR histograms rather than hadoop or yammer's
> -
>
> Key: HBASE-15222
> URL: https://issues.apache.org/jira/browse/HBASE-15222
> Project: HBase
> Issue Type: Bug
> Components: metrics
>Affects Versions: 2.0.0, 1.2.0, 1.3.0
>Reporter: Elliott Clark
>Assignee: Elliott Clark
> Fix For: 2.0.0, 1.3.0
>
>
> Running the benchmarks now, but it looks like the results are pretty extreme.
> The locking in our histograms is pretty extreme.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
