[jira] [Commented] (HBASE-5533) Add more metrics to HBase
[ https://issues.apache.org/jira/browse/HBASE-5533?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13428747#comment-13428747 ] Hudson commented on HBASE-5533: --- Integrated in HBase-0.94-security-on-Hadoop-23 #6 (See [https://builds.apache.org/job/HBase-0.94-security-on-Hadoop-23/6/]) HBASE-6377. HBASE-5533 metrics miss all operations submitted via MultiAction Applied 6377-0.94-remove-get-put-delete-histograms.patch (Revision 1361028) Result = FAILURE apurtell : Files : * /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java * /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/regionserver/metrics/RegionServerMetrics.java Add more metrics to HBase - Key: HBASE-5533 URL: https://issues.apache.org/jira/browse/HBASE-5533 Project: HBase Issue Type: Improvement Affects Versions: 0.92.2, 0.94.0 Reporter: Shaneal Manek Assignee: Shaneal Manek Priority: Minor Fix For: 0.92.2, 0.94.0, 0.96.0 Attachments: BlockingQueueContention.java, HBASE-5533-0.92-v4.patch, HBASE-5533-TRUNK-v6.patch, HBASE-5533-TRUNK-v6.patch, HBASE-5533-v7-0.92.patch, TimingOverhead.java, hbase-5533-0.92.patch, hbase5533-0.92-v2.patch, hbase5533-0.92-v3.patch, hbase5533-0.92-v5.patch, histogram_web_ui.png To debug/monitor production clusters, there are some more metrics I wish I had available. In particular: - Although the average FS latencies are useful, a 'histogram' of recent latencies (90% of reads completed in under 100ms, 99% in under 200ms, etc) would be more useful - Similar histograms of latencies on common operations (GET, PUT, DELETE) would be useful - Counting the number of accesses to each region to detect hotspotting - Exposing the current number of HLog files -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5533) Add more metrics to HBase
[ https://issues.apache.org/jira/browse/HBASE-5533?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13413673#comment-13413673 ] Hudson commented on HBASE-5533: --- Integrated in HBase-TRUNK-on-Hadoop-2.0.0 #92 (See [https://builds.apache.org/job/HBase-TRUNK-on-Hadoop-2.0.0/92/]) HBASE-6377. HBASE-5533 metrics miss all operations submitted via MultiAction Committed 6377-trunk-remove-get-put-delete-histograms.patch (Revision 1361026) Result = FAILURE apurtell : Files : * /hbase/trunk/hbase-server/src/main/jamon/org/apache/hadoop/hbase/tmpl/regionserver/ServerMetricsTmpl.jamon * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/metrics/RegionServerMetrics.java Add more metrics to HBase - Key: HBASE-5533 URL: https://issues.apache.org/jira/browse/HBASE-5533 Project: HBase Issue Type: Improvement Affects Versions: 0.92.2, 0.94.0 Reporter: Shaneal Manek Assignee: Shaneal Manek Priority: Minor Fix For: 0.92.2, 0.94.0, 0.96.0 Attachments: BlockingQueueContention.java, HBASE-5533-0.92-v4.patch, HBASE-5533-TRUNK-v6.patch, HBASE-5533-TRUNK-v6.patch, HBASE-5533-v7-0.92.patch, TimingOverhead.java, hbase-5533-0.92.patch, hbase5533-0.92-v2.patch, hbase5533-0.92-v3.patch, hbase5533-0.92-v5.patch, histogram_web_ui.png To debug/monitor production clusters, there are some more metrics I wish I had available. In particular: - Although the average FS latencies are useful, a 'histogram' of recent latencies (90% of reads completed in under 100ms, 99% in under 200ms, etc) would be more useful - Similar histograms of latencies on common operations (GET, PUT, DELETE) would be useful - Counting the number of accesses to each region to detect hotspotting - Exposing the current number of HLog files -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5533) Add more metrics to HBase
[ https://issues.apache.org/jira/browse/HBASE-5533?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13413416#comment-13413416 ] Hudson commented on HBASE-5533: --- Integrated in HBase-0.94-security #40 (See [https://builds.apache.org/job/HBase-0.94-security/40/]) HBASE-6377. HBASE-5533 metrics miss all operations submitted via MultiAction Applied 6377-0.94-remove-get-put-delete-histograms.patch (Revision 1361028) Result = SUCCESS apurtell : Files : * /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java * /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/regionserver/metrics/RegionServerMetrics.java Add more metrics to HBase - Key: HBASE-5533 URL: https://issues.apache.org/jira/browse/HBASE-5533 Project: HBase Issue Type: Improvement Affects Versions: 0.92.2, 0.94.0 Reporter: Shaneal Manek Assignee: Shaneal Manek Priority: Minor Fix For: 0.92.2, 0.94.0, 0.96.0 Attachments: BlockingQueueContention.java, HBASE-5533-0.92-v4.patch, HBASE-5533-TRUNK-v6.patch, HBASE-5533-TRUNK-v6.patch, HBASE-5533-v7-0.92.patch, TimingOverhead.java, hbase-5533-0.92.patch, hbase5533-0.92-v2.patch, hbase5533-0.92-v3.patch, hbase5533-0.92-v5.patch, histogram_web_ui.png To debug/monitor production clusters, there are some more metrics I wish I had available. In particular: - Although the average FS latencies are useful, a 'histogram' of recent latencies (90% of reads completed in under 100ms, 99% in under 200ms, etc) would be more useful - Similar histograms of latencies on common operations (GET, PUT, DELETE) would be useful - Counting the number of accesses to each region to detect hotspotting - Exposing the current number of HLog files -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5533) Add more metrics to HBase
[ https://issues.apache.org/jira/browse/HBASE-5533?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13413430#comment-13413430 ] Hudson commented on HBASE-5533: --- Integrated in HBase-0.94 #313 (See [https://builds.apache.org/job/HBase-0.94/313/]) HBASE-6377. HBASE-5533 metrics miss all operations submitted via MultiAction Applied 6377-0.94-remove-get-put-delete-histograms.patch (Revision 1361028) Result = FAILURE apurtell : Files : * /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java * /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/regionserver/metrics/RegionServerMetrics.java Add more metrics to HBase - Key: HBASE-5533 URL: https://issues.apache.org/jira/browse/HBASE-5533 Project: HBase Issue Type: Improvement Affects Versions: 0.92.2, 0.94.0 Reporter: Shaneal Manek Assignee: Shaneal Manek Priority: Minor Fix For: 0.92.2, 0.94.0, 0.96.0 Attachments: BlockingQueueContention.java, HBASE-5533-0.92-v4.patch, HBASE-5533-TRUNK-v6.patch, HBASE-5533-TRUNK-v6.patch, HBASE-5533-v7-0.92.patch, TimingOverhead.java, hbase-5533-0.92.patch, hbase5533-0.92-v2.patch, hbase5533-0.92-v3.patch, hbase5533-0.92-v5.patch, histogram_web_ui.png To debug/monitor production clusters, there are some more metrics I wish I had available. In particular: - Although the average FS latencies are useful, a 'histogram' of recent latencies (90% of reads completed in under 100ms, 99% in under 200ms, etc) would be more useful - Similar histograms of latencies on common operations (GET, PUT, DELETE) would be useful - Counting the number of accesses to each region to detect hotspotting - Exposing the current number of HLog files -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5533) Add more metrics to HBase
[ https://issues.apache.org/jira/browse/HBASE-5533?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13413434#comment-13413434 ] Hudson commented on HBASE-5533: --- Integrated in HBase-TRUNK #3122 (See [https://builds.apache.org/job/HBase-TRUNK/3122/]) HBASE-6377. HBASE-5533 metrics miss all operations submitted via MultiAction Committed 6377-trunk-remove-get-put-delete-histograms.patch (Revision 1361026) Result = FAILURE apurtell : Files : * /hbase/trunk/hbase-server/src/main/jamon/org/apache/hadoop/hbase/tmpl/regionserver/ServerMetricsTmpl.jamon * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/metrics/RegionServerMetrics.java Add more metrics to HBase - Key: HBASE-5533 URL: https://issues.apache.org/jira/browse/HBASE-5533 Project: HBase Issue Type: Improvement Affects Versions: 0.92.2, 0.94.0 Reporter: Shaneal Manek Assignee: Shaneal Manek Priority: Minor Fix For: 0.92.2, 0.94.0, 0.96.0 Attachments: BlockingQueueContention.java, HBASE-5533-0.92-v4.patch, HBASE-5533-TRUNK-v6.patch, HBASE-5533-TRUNK-v6.patch, HBASE-5533-v7-0.92.patch, TimingOverhead.java, hbase-5533-0.92.patch, hbase5533-0.92-v2.patch, hbase5533-0.92-v3.patch, hbase5533-0.92-v5.patch, histogram_web_ui.png To debug/monitor production clusters, there are some more metrics I wish I had available. In particular: - Although the average FS latencies are useful, a 'histogram' of recent latencies (90% of reads completed in under 100ms, 99% in under 200ms, etc) would be more useful - Similar histograms of latencies on common operations (GET, PUT, DELETE) would be useful - Counting the number of accesses to each region to detect hotspotting - Exposing the current number of HLog files -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5533) Add more metrics to HBase
[
https://issues.apache.org/jira/browse/HBASE-5533?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13291318#comment-13291318
]
Zhihong Ted Yu commented on HBASE-5533:
---
I saw a lot of the following in test output
(https://builds.apache.org/view/G-L/view/HBase/job/HBase-TRUNK/3000/testReport/org.apache.hadoop.hbase.regionserver/TestSplitTransactionOnCluster/testShutdownFixupWhenDaughterHasSplit/):
{code}
2012-06-07 18:33:14,623 ERROR
[RegionServer:0;juno.apache.org,39424,1339093992166]
util.MetricsDynamicMBeanBase(116): unknown metrics type:
org.apache.hadoop.hbase.metrics.histogram.MetricsHistogram
{code}
Add more metrics to HBase
-
Key: HBASE-5533
URL: https://issues.apache.org/jira/browse/HBASE-5533
Project: HBase
Issue Type: Improvement
Affects Versions: 0.92.2, 0.94.0
Reporter: Shaneal Manek
Assignee: Shaneal Manek
Priority: Minor
Fix For: 0.92.2, 0.94.0, 0.96.0
Attachments: BlockingQueueContention.java, HBASE-5533-0.92-v4.patch,
HBASE-5533-TRUNK-v6.patch, HBASE-5533-TRUNK-v6.patch,
HBASE-5533-v7-0.92.patch, TimingOverhead.java, hbase-5533-0.92.patch,
hbase5533-0.92-v2.patch, hbase5533-0.92-v3.patch, hbase5533-0.92-v5.patch,
histogram_web_ui.png
To debug/monitor production clusters, there are some more metrics I wish I
had available.
In particular:
- Although the average FS latencies are useful, a 'histogram' of recent
latencies (90% of reads completed in under 100ms, 99% in under 200ms, etc)
would be more useful
- Similar histograms of latencies on common operations (GET, PUT, DELETE)
would be useful
- Counting the number of accesses to each region to detect hotspotting
- Exposing the current number of HLog files
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5533) Add more metrics to HBase
[ https://issues.apache.org/jira/browse/HBASE-5533?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13288799#comment-13288799 ] Hudson commented on HBASE-5533: --- Integrated in HBase-0.92 #444 (See [https://builds.apache.org/job/HBase-0.92/444/]) HBASE-6131 Add attribution for code added by HBASE-5533 metrics (Revision 1346033) Result = FAILURE stack : Files : * /hbase/branches/0.92/pom.xml * /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/metrics/histogram/ExponentiallyDecayingSample.java * /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/metrics/histogram/MetricsHistogram.java * /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/metrics/histogram/Sample.java * /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/metrics/histogram/Snapshot.java * /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/metrics/histogram/UniformSample.java * /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/regionserver/metrics/RegionServerMetrics.java * /hbase/branches/0.92/src/test/java/org/apache/hadoop/hbase/metrics/TestExponentiallyDecayingSample.java * /hbase/branches/0.92/src/test/java/org/apache/hadoop/hbase/metrics/TestMetricsHistogram.java Add more metrics to HBase - Key: HBASE-5533 URL: https://issues.apache.org/jira/browse/HBASE-5533 Project: HBase Issue Type: Improvement Affects Versions: 0.92.2, 0.94.0 Reporter: Shaneal Manek Assignee: Shaneal Manek Priority: Minor Fix For: 0.92.2, 0.94.0, 0.96.0 Attachments: BlockingQueueContention.java, HBASE-5533-0.92-v4.patch, HBASE-5533-TRUNK-v6.patch, HBASE-5533-TRUNK-v6.patch, HBASE-5533-v7-0.92.patch, TimingOverhead.java, hbase-5533-0.92.patch, hbase5533-0.92-v2.patch, hbase5533-0.92-v3.patch, hbase5533-0.92-v5.patch, histogram_web_ui.png To debug/monitor production clusters, there are some more metrics I wish I had available. In particular: - Although the average FS latencies are useful, a 'histogram' of recent latencies (90% of reads completed in under 100ms, 99% in under 200ms, etc) would be more useful - Similar histograms of latencies on common operations (GET, PUT, DELETE) would be useful - Counting the number of accesses to each region to detect hotspotting - Exposing the current number of HLog files -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5533) Add more metrics to HBase
[ https://issues.apache.org/jira/browse/HBASE-5533?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13288849#comment-13288849 ] Hudson commented on HBASE-5533: --- Integrated in HBase-0.92-security #110 (See [https://builds.apache.org/job/HBase-0.92-security/110/]) HBASE-6131 Add attribution for code added by HBASE-5533 metrics (Revision 1346033) Result = FAILURE stack : Files : * /hbase/branches/0.92/pom.xml * /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/metrics/histogram/ExponentiallyDecayingSample.java * /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/metrics/histogram/MetricsHistogram.java * /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/metrics/histogram/Sample.java * /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/metrics/histogram/Snapshot.java * /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/metrics/histogram/UniformSample.java * /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/regionserver/metrics/RegionServerMetrics.java * /hbase/branches/0.92/src/test/java/org/apache/hadoop/hbase/metrics/TestExponentiallyDecayingSample.java * /hbase/branches/0.92/src/test/java/org/apache/hadoop/hbase/metrics/TestMetricsHistogram.java Add more metrics to HBase - Key: HBASE-5533 URL: https://issues.apache.org/jira/browse/HBASE-5533 Project: HBase Issue Type: Improvement Affects Versions: 0.92.2, 0.94.0 Reporter: Shaneal Manek Assignee: Shaneal Manek Priority: Minor Fix For: 0.92.2, 0.94.0, 0.96.0 Attachments: BlockingQueueContention.java, HBASE-5533-0.92-v4.patch, HBASE-5533-TRUNK-v6.patch, HBASE-5533-TRUNK-v6.patch, HBASE-5533-v7-0.92.patch, TimingOverhead.java, hbase-5533-0.92.patch, hbase5533-0.92-v2.patch, hbase5533-0.92-v3.patch, hbase5533-0.92-v5.patch, histogram_web_ui.png To debug/monitor production clusters, there are some more metrics I wish I had available. In particular: - Although the average FS latencies are useful, a 'histogram' of recent latencies (90% of reads completed in under 100ms, 99% in under 200ms, etc) would be more useful - Similar histograms of latencies on common operations (GET, PUT, DELETE) would be useful - Counting the number of accesses to each region to detect hotspotting - Exposing the current number of HLog files -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5533) Add more metrics to HBase
[ https://issues.apache.org/jira/browse/HBASE-5533?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13287186#comment-13287186 ] Hudson commented on HBASE-5533: --- Integrated in HBase-0.94-security #33 (See [https://builds.apache.org/job/HBase-0.94-security/33/]) HBASE-6131 Add attribution for code added by HBASE-5533 metrics (Revision 1344305) HBASE-6131 Add attribution for code added by HBASE-5533 metrics (Revision 1344304) Result = FAILURE stack : Files : * /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/metrics/histogram/ExponentiallyDecayingSample.java * /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/metrics/histogram/Sample.java * /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/metrics/histogram/Snapshot.java * /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/metrics/histogram/UniformSample.java stack : Files : * /hbase/branches/0.94/pom.xml * /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/metrics/histogram/ExponentiallyDecayingSample.java * /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/metrics/histogram/MetricsHistogram.java * /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/metrics/histogram/Sample.java * /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/metrics/histogram/Snapshot.java * /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/metrics/histogram/UniformSample.java * /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/regionserver/metrics/RegionServerMetrics.java * /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/metrics/TestExponentiallyDecayingSample.java * /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/metrics/TestMetricsHistogram.java Add more metrics to HBase - Key: HBASE-5533 URL: https://issues.apache.org/jira/browse/HBASE-5533 Project: HBase Issue Type: Improvement Affects Versions: 0.92.2, 0.94.0 Reporter: Shaneal Manek Assignee: Shaneal Manek Priority: Minor Fix For: 0.92.2, 0.94.0, 0.96.0 Attachments: BlockingQueueContention.java, HBASE-5533-0.92-v4.patch, HBASE-5533-TRUNK-v6.patch, HBASE-5533-TRUNK-v6.patch, HBASE-5533-v7-0.92.patch, TimingOverhead.java, hbase-5533-0.92.patch, hbase5533-0.92-v2.patch, hbase5533-0.92-v3.patch, hbase5533-0.92-v5.patch, histogram_web_ui.png To debug/monitor production clusters, there are some more metrics I wish I had available. In particular: - Although the average FS latencies are useful, a 'histogram' of recent latencies (90% of reads completed in under 100ms, 99% in under 200ms, etc) would be more useful - Similar histograms of latencies on common operations (GET, PUT, DELETE) would be useful - Counting the number of accesses to each region to detect hotspotting - Exposing the current number of HLog files -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5533) Add more metrics to HBase
[ https://issues.apache.org/jira/browse/HBASE-5533?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13285807#comment-13285807 ] Hudson commented on HBASE-5533: --- Integrated in HBase-TRUNK #2953 (See [https://builds.apache.org/job/HBase-TRUNK/2953/]) HBASE-6131 Add attribution for code added by HBASE-5533 metrics (Revision 1344299) Result = FAILURE stack : Files : * /hbase/trunk/hbase-server/pom.xml * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/metrics/histogram/ExponentiallyDecayingSample.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/metrics/histogram/MetricsHistogram.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/metrics/histogram/Sample.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/metrics/histogram/Snapshot.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/metrics/histogram/UniformSample.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/metrics/RegionServerMetrics.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/metrics/TestExponentiallyDecayingSample.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/metrics/TestMetricsHistogram.java * /hbase/trunk/pom.xml Add more metrics to HBase - Key: HBASE-5533 URL: https://issues.apache.org/jira/browse/HBASE-5533 Project: HBase Issue Type: Improvement Affects Versions: 0.92.2, 0.94.0 Reporter: Shaneal Manek Assignee: Shaneal Manek Priority: Minor Fix For: 0.92.2, 0.94.0, 0.96.0 Attachments: BlockingQueueContention.java, HBASE-5533-0.92-v4.patch, HBASE-5533-TRUNK-v6.patch, HBASE-5533-TRUNK-v6.patch, HBASE-5533-v7-0.92.patch, TimingOverhead.java, hbase-5533-0.92.patch, hbase5533-0.92-v2.patch, hbase5533-0.92-v3.patch, hbase5533-0.92-v5.patch, histogram_web_ui.png To debug/monitor production clusters, there are some more metrics I wish I had available. In particular: - Although the average FS latencies are useful, a 'histogram' of recent latencies (90% of reads completed in under 100ms, 99% in under 200ms, etc) would be more useful - Similar histograms of latencies on common operations (GET, PUT, DELETE) would be useful - Counting the number of accesses to each region to detect hotspotting - Exposing the current number of HLog files -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5533) Add more metrics to HBase
[ https://issues.apache.org/jira/browse/HBASE-5533?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13285813#comment-13285813 ] Hudson commented on HBASE-5533: --- Integrated in HBase-0.94 #231 (See [https://builds.apache.org/job/HBase-0.94/231/]) HBASE-6131 Add attribution for code added by HBASE-5533 metrics (Revision 1344305) HBASE-6131 Add attribution for code added by HBASE-5533 metrics (Revision 1344304) Result = SUCCESS stack : Files : * /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/metrics/histogram/ExponentiallyDecayingSample.java * /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/metrics/histogram/Sample.java * /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/metrics/histogram/Snapshot.java * /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/metrics/histogram/UniformSample.java stack : Files : * /hbase/branches/0.94/pom.xml * /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/metrics/histogram/ExponentiallyDecayingSample.java * /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/metrics/histogram/MetricsHistogram.java * /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/metrics/histogram/Sample.java * /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/metrics/histogram/Snapshot.java * /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/metrics/histogram/UniformSample.java * /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/regionserver/metrics/RegionServerMetrics.java * /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/metrics/TestExponentiallyDecayingSample.java * /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/metrics/TestMetricsHistogram.java Add more metrics to HBase - Key: HBASE-5533 URL: https://issues.apache.org/jira/browse/HBASE-5533 Project: HBase Issue Type: Improvement Affects Versions: 0.92.2, 0.94.0 Reporter: Shaneal Manek Assignee: Shaneal Manek Priority: Minor Fix For: 0.92.2, 0.94.0, 0.96.0 Attachments: BlockingQueueContention.java, HBASE-5533-0.92-v4.patch, HBASE-5533-TRUNK-v6.patch, HBASE-5533-TRUNK-v6.patch, HBASE-5533-v7-0.92.patch, TimingOverhead.java, hbase-5533-0.92.patch, hbase5533-0.92-v2.patch, hbase5533-0.92-v3.patch, hbase5533-0.92-v5.patch, histogram_web_ui.png To debug/monitor production clusters, there are some more metrics I wish I had available. In particular: - Although the average FS latencies are useful, a 'histogram' of recent latencies (90% of reads completed in under 100ms, 99% in under 200ms, etc) would be more useful - Similar histograms of latencies on common operations (GET, PUT, DELETE) would be useful - Counting the number of accesses to each region to detect hotspotting - Exposing the current number of HLog files -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5533) Add more metrics to HBase
[ https://issues.apache.org/jira/browse/HBASE-5533?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13286210#comment-13286210 ] Hudson commented on HBASE-5533: --- Integrated in HBase-TRUNK-on-Hadoop-2.0.0 #33 (See [https://builds.apache.org/job/HBase-TRUNK-on-Hadoop-2.0.0/33/]) HBASE-6131 Add attribution for code added by HBASE-5533 metrics (Revision 1344299) Result = FAILURE stack : Files : * /hbase/trunk/hbase-server/pom.xml * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/metrics/histogram/ExponentiallyDecayingSample.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/metrics/histogram/MetricsHistogram.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/metrics/histogram/Sample.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/metrics/histogram/Snapshot.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/metrics/histogram/UniformSample.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/metrics/RegionServerMetrics.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/metrics/TestExponentiallyDecayingSample.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/metrics/TestMetricsHistogram.java * /hbase/trunk/pom.xml Add more metrics to HBase - Key: HBASE-5533 URL: https://issues.apache.org/jira/browse/HBASE-5533 Project: HBase Issue Type: Improvement Affects Versions: 0.92.2, 0.94.0 Reporter: Shaneal Manek Assignee: Shaneal Manek Priority: Minor Fix For: 0.92.2, 0.94.0, 0.96.0 Attachments: BlockingQueueContention.java, HBASE-5533-0.92-v4.patch, HBASE-5533-TRUNK-v6.patch, HBASE-5533-TRUNK-v6.patch, HBASE-5533-v7-0.92.patch, TimingOverhead.java, hbase-5533-0.92.patch, hbase5533-0.92-v2.patch, hbase5533-0.92-v3.patch, hbase5533-0.92-v5.patch, histogram_web_ui.png To debug/monitor production clusters, there are some more metrics I wish I had available. In particular: - Although the average FS latencies are useful, a 'histogram' of recent latencies (90% of reads completed in under 100ms, 99% in under 200ms, etc) would be more useful - Similar histograms of latencies on common operations (GET, PUT, DELETE) would be useful - Counting the number of accesses to each region to detect hotspotting - Exposing the current number of HLog files -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5533) Add more metrics to HBase
[ https://issues.apache.org/jira/browse/HBASE-5533?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13285262#comment-13285262 ] stack commented on HBASE-5533: -- @Ted Thanks for noticing. I made a blocker over on HBASE-6131 to fix. Add more metrics to HBase - Key: HBASE-5533 URL: https://issues.apache.org/jira/browse/HBASE-5533 Project: HBase Issue Type: Improvement Affects Versions: 0.92.2, 0.94.0 Reporter: Shaneal Manek Assignee: Shaneal Manek Priority: Minor Fix For: 0.92.2, 0.94.0, 0.96.0 Attachments: BlockingQueueContention.java, HBASE-5533-0.92-v4.patch, HBASE-5533-TRUNK-v6.patch, HBASE-5533-TRUNK-v6.patch, HBASE-5533-v7-0.92.patch, TimingOverhead.java, hbase-5533-0.92.patch, hbase5533-0.92-v2.patch, hbase5533-0.92-v3.patch, hbase5533-0.92-v5.patch, histogram_web_ui.png To debug/monitor production clusters, there are some more metrics I wish I had available. In particular: - Although the average FS latencies are useful, a 'histogram' of recent latencies (90% of reads completed in under 100ms, 99% in under 200ms, etc) would be more useful - Similar histograms of latencies on common operations (GET, PUT, DELETE) would be useful - Counting the number of accesses to each region to detect hotspotting - Exposing the current number of HLog files -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5533) Add more metrics to HBase
[ https://issues.apache.org/jira/browse/HBASE-5533?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13283920#comment-13283920 ] Ted Nyman commented on HBASE-5533: -- Unless I'm missing something (and I hope I am), this addition is completely without proper attribution. The code is copied virtually line for line from Coda Hale's metrics library, in particuar: https://github.com/codahale/metrics/tree/master/metrics-core/src/main/java/com/yammer/metrics/stats. This is the exact same implementation (even comments) as in https://github.com/apache/hbase/tree/trunk/src/main/java/org/apache/hadoop/hbase/metrics/histogram. There is no credit in hbase license, inline comments, or in the source code itself. Add more metrics to HBase - Key: HBASE-5533 URL: https://issues.apache.org/jira/browse/HBASE-5533 Project: HBase Issue Type: Improvement Affects Versions: 0.92.2, 0.94.0 Reporter: Shaneal Manek Assignee: Shaneal Manek Priority: Minor Fix For: 0.92.2, 0.94.0, 0.96.0 Attachments: BlockingQueueContention.java, HBASE-5533-0.92-v4.patch, HBASE-5533-TRUNK-v6.patch, HBASE-5533-TRUNK-v6.patch, HBASE-5533-v7-0.92.patch, TimingOverhead.java, hbase-5533-0.92.patch, hbase5533-0.92-v2.patch, hbase5533-0.92-v3.patch, hbase5533-0.92-v5.patch, histogram_web_ui.png To debug/monitor production clusters, there are some more metrics I wish I had available. In particular: - Although the average FS latencies are useful, a 'histogram' of recent latencies (90% of reads completed in under 100ms, 99% in under 200ms, etc) would be more useful - Similar histograms of latencies on common operations (GET, PUT, DELETE) would be useful - Counting the number of accesses to each region to detect hotspotting - Exposing the current number of HLog files -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5533) Add more metrics to HBase
[ https://issues.apache.org/jira/browse/HBASE-5533?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13283923#comment-13283923 ] Jeff Hodges commented on HBASE-5533: The original implementation in metrics uses the Apache license, but the copyright is to Coda Hale and Yammer, Inc.. Add more metrics to HBase - Key: HBASE-5533 URL: https://issues.apache.org/jira/browse/HBASE-5533 Project: HBase Issue Type: Improvement Affects Versions: 0.92.2, 0.94.0 Reporter: Shaneal Manek Assignee: Shaneal Manek Priority: Minor Fix For: 0.92.2, 0.94.0, 0.96.0 Attachments: BlockingQueueContention.java, HBASE-5533-0.92-v4.patch, HBASE-5533-TRUNK-v6.patch, HBASE-5533-TRUNK-v6.patch, HBASE-5533-v7-0.92.patch, TimingOverhead.java, hbase-5533-0.92.patch, hbase5533-0.92-v2.patch, hbase5533-0.92-v3.patch, hbase5533-0.92-v5.patch, histogram_web_ui.png To debug/monitor production clusters, there are some more metrics I wish I had available. In particular: - Although the average FS latencies are useful, a 'histogram' of recent latencies (90% of reads completed in under 100ms, 99% in under 200ms, etc) would be more useful - Similar histograms of latencies on common operations (GET, PUT, DELETE) would be useful - Counting the number of accesses to each region to detect hotspotting - Exposing the current number of HLog files -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5533) Add more metrics to HBase
[ https://issues.apache.org/jira/browse/HBASE-5533?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13246556#comment-13246556 ] Hudson commented on HBASE-5533: --- Integrated in HBase-0.92 #353 (See [https://builds.apache.org/job/HBase-0.92/353/]) HBASE-5533 Add more metrics to HBase (Revision 1309466) Result = FAILURE stack : Files : * /hbase/branches/0.92/CHANGES.txt * /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/io/hfile/HFile.java * /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileReaderV1.java * /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileReaderV2.java * /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileWriterV1.java * /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileWriterV2.java * /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/metrics/ExactCounterMetric.java * /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/metrics/histogram * /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/metrics/histogram/ExponentiallyDecayingSample.java * /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/metrics/histogram/MetricsHistogram.java * /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/metrics/histogram/Sample.java * /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/metrics/histogram/Snapshot.java * /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/metrics/histogram/UniformSample.java * /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java * /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/regionserver/metrics/RegionServerMetrics.java * /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLog.java * /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/util/Threads.java * /hbase/branches/0.92/src/test/java/org/apache/hadoop/hbase/metrics/TestExactCounterMetric.java * /hbase/branches/0.92/src/test/java/org/apache/hadoop/hbase/metrics/TestExponentiallyDecayingSample.java * /hbase/branches/0.92/src/test/java/org/apache/hadoop/hbase/metrics/TestMetricsHistogram.java Add more metrics to HBase - Key: HBASE-5533 URL: https://issues.apache.org/jira/browse/HBASE-5533 Project: HBase Issue Type: Improvement Affects Versions: 0.92.2, 0.94.0 Reporter: Shaneal Manek Assignee: Shaneal Manek Priority: Minor Fix For: 0.92.2, 0.94.0, 0.96.0 Attachments: BlockingQueueContention.java, HBASE-5533-0.92-v4.patch, HBASE-5533-TRUNK-v6.patch, HBASE-5533-TRUNK-v6.patch, HBASE-5533-v7-0.92.patch, TimingOverhead.java, hbase-5533-0.92.patch, hbase5533-0.92-v2.patch, hbase5533-0.92-v3.patch, hbase5533-0.92-v5.patch, histogram_web_ui.png To debug/monitor production clusters, there are some more metrics I wish I had available. In particular: - Although the average FS latencies are useful, a 'histogram' of recent latencies (90% of reads completed in under 100ms, 99% in under 200ms, etc) would be more useful - Similar histograms of latencies on common operations (GET, PUT, DELETE) would be useful - Counting the number of accesses to each region to detect hotspotting - Exposing the current number of HLog files -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5533) Add more metrics to HBase
[ https://issues.apache.org/jira/browse/HBASE-5533?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13246961#comment-13246961 ] Hudson commented on HBASE-5533: --- Integrated in HBase-0.92-security #104 (See [https://builds.apache.org/job/HBase-0.92-security/104/]) HBASE-5533 Add more metrics to HBase (Revision 1309466) Result = FAILURE stack : Files : * /hbase/branches/0.92/CHANGES.txt * /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/io/hfile/HFile.java * /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileReaderV1.java * /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileReaderV2.java * /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileWriterV1.java * /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileWriterV2.java * /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/metrics/ExactCounterMetric.java * /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/metrics/histogram * /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/metrics/histogram/ExponentiallyDecayingSample.java * /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/metrics/histogram/MetricsHistogram.java * /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/metrics/histogram/Sample.java * /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/metrics/histogram/Snapshot.java * /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/metrics/histogram/UniformSample.java * /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java * /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/regionserver/metrics/RegionServerMetrics.java * /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLog.java * /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/util/Threads.java * /hbase/branches/0.92/src/test/java/org/apache/hadoop/hbase/metrics/TestExactCounterMetric.java * /hbase/branches/0.92/src/test/java/org/apache/hadoop/hbase/metrics/TestExponentiallyDecayingSample.java * /hbase/branches/0.92/src/test/java/org/apache/hadoop/hbase/metrics/TestMetricsHistogram.java Add more metrics to HBase - Key: HBASE-5533 URL: https://issues.apache.org/jira/browse/HBASE-5533 Project: HBase Issue Type: Improvement Affects Versions: 0.92.2, 0.94.0 Reporter: Shaneal Manek Assignee: Shaneal Manek Priority: Minor Fix For: 0.92.2, 0.94.0, 0.96.0 Attachments: BlockingQueueContention.java, HBASE-5533-0.92-v4.patch, HBASE-5533-TRUNK-v6.patch, HBASE-5533-TRUNK-v6.patch, HBASE-5533-v7-0.92.patch, TimingOverhead.java, hbase-5533-0.92.patch, hbase5533-0.92-v2.patch, hbase5533-0.92-v3.patch, hbase5533-0.92-v5.patch, histogram_web_ui.png To debug/monitor production clusters, there are some more metrics I wish I had available. In particular: - Although the average FS latencies are useful, a 'histogram' of recent latencies (90% of reads completed in under 100ms, 99% in under 200ms, etc) would be more useful - Similar histograms of latencies on common operations (GET, PUT, DELETE) would be useful - Counting the number of accesses to each region to detect hotspotting - Exposing the current number of HLog files -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5533) Add more metrics to HBase
[ https://issues.apache.org/jira/browse/HBASE-5533?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13240893#comment-13240893 ] Hudson commented on HBASE-5533: --- Integrated in HBase-TRUNK #2697 (See [https://builds.apache.org/job/HBase-TRUNK/2697/]) HBASE-5533 Add more metrics to HBase (Revision 1305499) Result = FAILURE stack : Files : * /hbase/trunk/src/main/java/org/apache/hadoop/hbase/io/hfile/HFile.java * /hbase/trunk/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileReaderV1.java * /hbase/trunk/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileReaderV2.java * /hbase/trunk/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileWriterV1.java * /hbase/trunk/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileWriterV2.java * /hbase/trunk/src/main/java/org/apache/hadoop/hbase/metrics/ExactCounterMetric.java * /hbase/trunk/src/main/java/org/apache/hadoop/hbase/metrics/histogram * /hbase/trunk/src/main/java/org/apache/hadoop/hbase/metrics/histogram/ExponentiallyDecayingSample.java * /hbase/trunk/src/main/java/org/apache/hadoop/hbase/metrics/histogram/MetricsHistogram.java * /hbase/trunk/src/main/java/org/apache/hadoop/hbase/metrics/histogram/Sample.java * /hbase/trunk/src/main/java/org/apache/hadoop/hbase/metrics/histogram/Snapshot.java * /hbase/trunk/src/main/java/org/apache/hadoop/hbase/metrics/histogram/UniformSample.java * /hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java * /hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/metrics/RegionServerMetrics.java * /hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLog.java * /hbase/trunk/src/main/java/org/apache/hadoop/hbase/util/Threads.java * /hbase/trunk/src/test/java/org/apache/hadoop/hbase/metrics/TestExactCounterMetric.java * /hbase/trunk/src/test/java/org/apache/hadoop/hbase/metrics/TestExponentiallyDecayingSample.java * /hbase/trunk/src/test/java/org/apache/hadoop/hbase/metrics/TestMetricsHistogram.java Add more metrics to HBase - Key: HBASE-5533 URL: https://issues.apache.org/jira/browse/HBASE-5533 Project: HBase Issue Type: Improvement Affects Versions: 0.92.2, 0.94.0 Reporter: Shaneal Manek Assignee: Shaneal Manek Priority: Minor Fix For: 0.94.0, 0.96.0 Attachments: BlockingQueueContention.java, HBASE-5533-0.92-v4.patch, HBASE-5533-TRUNK-v6.patch, HBASE-5533-TRUNK-v6.patch, TimingOverhead.java, hbase-5533-0.92.patch, hbase5533-0.92-v2.patch, hbase5533-0.92-v3.patch, hbase5533-0.92-v5.patch, histogram_web_ui.png To debug/monitor production clusters, there are some more metrics I wish I had available. In particular: - Although the average FS latencies are useful, a 'histogram' of recent latencies (90% of reads completed in under 100ms, 99% in under 200ms, etc) would be more useful - Similar histograms of latencies on common operations (GET, PUT, DELETE) would be useful - Counting the number of accesses to each region to detect hotspotting - Exposing the current number of HLog files -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5533) Add more metrics to HBase
[ https://issues.apache.org/jira/browse/HBASE-5533?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13239212#comment-13239212 ] Hudson commented on HBASE-5533: --- Integrated in HBase-TRUNK-security #151 (See [https://builds.apache.org/job/HBase-TRUNK-security/151/]) HBASE-5533 Add more metrics to HBase (Revision 1305499) Result = FAILURE stack : Files : * /hbase/trunk/src/main/java/org/apache/hadoop/hbase/io/hfile/HFile.java * /hbase/trunk/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileReaderV1.java * /hbase/trunk/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileReaderV2.java * /hbase/trunk/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileWriterV1.java * /hbase/trunk/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileWriterV2.java * /hbase/trunk/src/main/java/org/apache/hadoop/hbase/metrics/ExactCounterMetric.java * /hbase/trunk/src/main/java/org/apache/hadoop/hbase/metrics/histogram * /hbase/trunk/src/main/java/org/apache/hadoop/hbase/metrics/histogram/ExponentiallyDecayingSample.java * /hbase/trunk/src/main/java/org/apache/hadoop/hbase/metrics/histogram/MetricsHistogram.java * /hbase/trunk/src/main/java/org/apache/hadoop/hbase/metrics/histogram/Sample.java * /hbase/trunk/src/main/java/org/apache/hadoop/hbase/metrics/histogram/Snapshot.java * /hbase/trunk/src/main/java/org/apache/hadoop/hbase/metrics/histogram/UniformSample.java * /hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java * /hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/metrics/RegionServerMetrics.java * /hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLog.java * /hbase/trunk/src/main/java/org/apache/hadoop/hbase/util/Threads.java * /hbase/trunk/src/test/java/org/apache/hadoop/hbase/metrics/TestExactCounterMetric.java * /hbase/trunk/src/test/java/org/apache/hadoop/hbase/metrics/TestExponentiallyDecayingSample.java * /hbase/trunk/src/test/java/org/apache/hadoop/hbase/metrics/TestMetricsHistogram.java Add more metrics to HBase - Key: HBASE-5533 URL: https://issues.apache.org/jira/browse/HBASE-5533 Project: HBase Issue Type: Improvement Affects Versions: 0.92.2, 0.94.0 Reporter: Shaneal Manek Assignee: Shaneal Manek Priority: Minor Fix For: 0.94.0, 0.96.0 Attachments: BlockingQueueContention.java, HBASE-5533-0.92-v4.patch, HBASE-5533-TRUNK-v6.patch, HBASE-5533-TRUNK-v6.patch, TimingOverhead.java, hbase-5533-0.92.patch, hbase5533-0.92-v2.patch, hbase5533-0.92-v3.patch, hbase5533-0.92-v5.patch, histogram_web_ui.png To debug/monitor production clusters, there are some more metrics I wish I had available. In particular: - Although the average FS latencies are useful, a 'histogram' of recent latencies (90% of reads completed in under 100ms, 99% in under 200ms, etc) would be more useful - Similar histograms of latencies on common operations (GET, PUT, DELETE) would be useful - Counting the number of accesses to each region to detect hotspotting - Exposing the current number of HLog files -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5533) Add more metrics to HBase
[ https://issues.apache.org/jira/browse/HBASE-5533?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13240009#comment-13240009 ] Shaneal Manek commented on HBASE-5533: -- Thanks for the fix Lars (HBASE-5641, for future reference). I'll update the patch for 0.92 ASAP. Add more metrics to HBase - Key: HBASE-5533 URL: https://issues.apache.org/jira/browse/HBASE-5533 Project: HBase Issue Type: Improvement Affects Versions: 0.92.2, 0.94.0 Reporter: Shaneal Manek Assignee: Shaneal Manek Priority: Minor Fix For: 0.94.0, 0.96.0 Attachments: BlockingQueueContention.java, HBASE-5533-0.92-v4.patch, HBASE-5533-TRUNK-v6.patch, HBASE-5533-TRUNK-v6.patch, TimingOverhead.java, hbase-5533-0.92.patch, hbase5533-0.92-v2.patch, hbase5533-0.92-v3.patch, hbase5533-0.92-v5.patch, histogram_web_ui.png To debug/monitor production clusters, there are some more metrics I wish I had available. In particular: - Although the average FS latencies are useful, a 'histogram' of recent latencies (90% of reads completed in under 100ms, 99% in under 200ms, etc) would be more useful - Similar histograms of latencies on common operations (GET, PUT, DELETE) would be useful - Counting the number of accesses to each region to detect hotspotting - Exposing the current number of HLog files -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5533) Add more metrics to HBase
[ https://issues.apache.org/jira/browse/HBASE-5533?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13238647#comment-13238647 ] Hadoop QA commented on HBASE-5533: -- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12519846/HBASE-5533-TRUNK-v6.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 6 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. -1 findbugs. The patch appears to introduce 1 new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests: org.apache.hadoop.hbase.regionserver.TestSplitTransactionOnCluster org.apache.hadoop.hbase.client.TestFromClientSide Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/1307//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/1307//artifact/trunk/patchprocess/newPatchFindbugsWarnings.html Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/1307//console This message is automatically generated. Add more metrics to HBase - Key: HBASE-5533 URL: https://issues.apache.org/jira/browse/HBASE-5533 Project: HBase Issue Type: Improvement Affects Versions: 0.92.2, 0.94.0 Reporter: Shaneal Manek Assignee: Shaneal Manek Priority: Minor Attachments: BlockingQueueContention.java, HBASE-5533-0.92-v4.patch, HBASE-5533-TRUNK-v6.patch, HBASE-5533-TRUNK-v6.patch, TimingOverhead.java, hbase-5533-0.92.patch, hbase5533-0.92-v2.patch, hbase5533-0.92-v3.patch, hbase5533-0.92-v5.patch, histogram_web_ui.png To debug/monitor production clusters, there are some more metrics I wish I had available. In particular: - Although the average FS latencies are useful, a 'histogram' of recent latencies (90% of reads completed in under 100ms, 99% in under 200ms, etc) would be more useful - Similar histograms of latencies on common operations (GET, PUT, DELETE) would be useful - Counting the number of accesses to each region to detect hotspotting - Exposing the current number of HLog files -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5533) Add more metrics to HBase
[ https://issues.apache.org/jira/browse/HBASE-5533?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13238817#comment-13238817 ] Lars Hofhansl commented on HBASE-5533: -- Committed to 0.94 as well. Add more metrics to HBase - Key: HBASE-5533 URL: https://issues.apache.org/jira/browse/HBASE-5533 Project: HBase Issue Type: Improvement Affects Versions: 0.92.2, 0.94.0 Reporter: Shaneal Manek Assignee: Shaneal Manek Priority: Minor Fix For: 0.94.0, 0.96.0 Attachments: BlockingQueueContention.java, HBASE-5533-0.92-v4.patch, HBASE-5533-TRUNK-v6.patch, HBASE-5533-TRUNK-v6.patch, TimingOverhead.java, hbase-5533-0.92.patch, hbase5533-0.92-v2.patch, hbase5533-0.92-v3.patch, hbase5533-0.92-v5.patch, histogram_web_ui.png To debug/monitor production clusters, there are some more metrics I wish I had available. In particular: - Although the average FS latencies are useful, a 'histogram' of recent latencies (90% of reads completed in under 100ms, 99% in under 200ms, etc) would be more useful - Similar histograms of latencies on common operations (GET, PUT, DELETE) would be useful - Counting the number of accesses to each region to detect hotspotting - Exposing the current number of HLog files -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5533) Add more metrics to HBase
[ https://issues.apache.org/jira/browse/HBASE-5533?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13239000#comment-13239000 ] Hudson commented on HBASE-5533: --- Integrated in HBase-0.94-security #3 (See [https://builds.apache.org/job/HBase-0.94-security/3/]) HBASE-5533 Add more metrics to HBase (Shaneal Manek) (Revision 1305582) Result = ABORTED larsh : Files : * /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/io/hfile/HFile.java * /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileReaderV1.java * /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileReaderV2.java * /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileWriterV1.java * /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileWriterV2.java * /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/metrics/ExactCounterMetric.java * /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/metrics/histogram * /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/metrics/histogram/ExponentiallyDecayingSample.java * /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/metrics/histogram/MetricsHistogram.java * /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/metrics/histogram/Sample.java * /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/metrics/histogram/Snapshot.java * /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/metrics/histogram/UniformSample.java * /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java * /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/regionserver/metrics/RegionServerMetrics.java * /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLog.java * /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/util/Threads.java * /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/metrics/TestExactCounterMetric.java * /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/metrics/TestExponentiallyDecayingSample.java * /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/metrics/TestMetricsHistogram.java Add more metrics to HBase - Key: HBASE-5533 URL: https://issues.apache.org/jira/browse/HBASE-5533 Project: HBase Issue Type: Improvement Affects Versions: 0.92.2, 0.94.0 Reporter: Shaneal Manek Assignee: Shaneal Manek Priority: Minor Fix For: 0.94.0, 0.96.0 Attachments: BlockingQueueContention.java, HBASE-5533-0.92-v4.patch, HBASE-5533-TRUNK-v6.patch, HBASE-5533-TRUNK-v6.patch, TimingOverhead.java, hbase-5533-0.92.patch, hbase5533-0.92-v2.patch, hbase5533-0.92-v3.patch, hbase5533-0.92-v5.patch, histogram_web_ui.png To debug/monitor production clusters, there are some more metrics I wish I had available. In particular: - Although the average FS latencies are useful, a 'histogram' of recent latencies (90% of reads completed in under 100ms, 99% in under 200ms, etc) would be more useful - Similar histograms of latencies on common operations (GET, PUT, DELETE) would be useful - Counting the number of accesses to each region to detect hotspotting - Exposing the current number of HLog files -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5533) Add more metrics to HBase
[ https://issues.apache.org/jira/browse/HBASE-5533?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13239006#comment-13239006 ] Hudson commented on HBASE-5533: --- Integrated in HBase-0.94 #58 (See [https://builds.apache.org/job/HBase-0.94/58/]) HBASE-5533 Add more metrics to HBase (Shaneal Manek) (Revision 1305582) Result = ABORTED larsh : Files : * /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/io/hfile/HFile.java * /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileReaderV1.java * /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileReaderV2.java * /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileWriterV1.java * /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileWriterV2.java * /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/metrics/ExactCounterMetric.java * /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/metrics/histogram * /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/metrics/histogram/ExponentiallyDecayingSample.java * /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/metrics/histogram/MetricsHistogram.java * /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/metrics/histogram/Sample.java * /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/metrics/histogram/Snapshot.java * /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/metrics/histogram/UniformSample.java * /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java * /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/regionserver/metrics/RegionServerMetrics.java * /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLog.java * /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/util/Threads.java * /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/metrics/TestExactCounterMetric.java * /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/metrics/TestExponentiallyDecayingSample.java * /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/metrics/TestMetricsHistogram.java Add more metrics to HBase - Key: HBASE-5533 URL: https://issues.apache.org/jira/browse/HBASE-5533 Project: HBase Issue Type: Improvement Affects Versions: 0.92.2, 0.94.0 Reporter: Shaneal Manek Assignee: Shaneal Manek Priority: Minor Fix For: 0.94.0, 0.96.0 Attachments: BlockingQueueContention.java, HBASE-5533-0.92-v4.patch, HBASE-5533-TRUNK-v6.patch, HBASE-5533-TRUNK-v6.patch, TimingOverhead.java, hbase-5533-0.92.patch, hbase5533-0.92-v2.patch, hbase5533-0.92-v3.patch, hbase5533-0.92-v5.patch, histogram_web_ui.png To debug/monitor production clusters, there are some more metrics I wish I had available. In particular: - Although the average FS latencies are useful, a 'histogram' of recent latencies (90% of reads completed in under 100ms, 99% in under 200ms, etc) would be more useful - Similar histograms of latencies on common operations (GET, PUT, DELETE) would be useful - Counting the number of accesses to each region to detect hotspotting - Exposing the current number of HLog files -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5533) Add more metrics to HBase
[
https://issues.apache.org/jira/browse/HBASE-5533?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13239127#comment-13239127
]
Lars Hofhansl commented on HBASE-5533:
--
While testing a 0.94.0RC I found that this is preventing HBase from shutting
down.
{code}
decayingSampleTick1 prio=10 tid=0x7faf5c047000 nid=0x216c waiting on condi
tion [0x7faf345c1000]
java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for 0xb853e308 (a java.util.concurrent.lock
s.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226
)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject
.awaitNanos(AbstractQueuedSynchronizer.java:2081)
at java.util.concurrent.DelayQueue.take(DelayQueue.java:193)
at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.tak
e(ScheduledThreadPoolExecutor.java:688)
at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.tak
e(ScheduledThreadPoolExecutor.java:681)
at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1043)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1103)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:679)
{code}
This is the only non-daemon thread I found.
Add more metrics to HBase
-
Key: HBASE-5533
URL: https://issues.apache.org/jira/browse/HBASE-5533
Project: HBase
Issue Type: Improvement
Affects Versions: 0.92.2, 0.94.0
Reporter: Shaneal Manek
Assignee: Shaneal Manek
Priority: Minor
Fix For: 0.94.0, 0.96.0
Attachments: BlockingQueueContention.java, HBASE-5533-0.92-v4.patch,
HBASE-5533-TRUNK-v6.patch, HBASE-5533-TRUNK-v6.patch, TimingOverhead.java,
hbase-5533-0.92.patch, hbase5533-0.92-v2.patch, hbase5533-0.92-v3.patch,
hbase5533-0.92-v5.patch, histogram_web_ui.png
To debug/monitor production clusters, there are some more metrics I wish I
had available.
In particular:
- Although the average FS latencies are useful, a 'histogram' of recent
latencies (90% of reads completed in under 100ms, 99% in under 200ms, etc)
would be more useful
- Similar histograms of latencies on common operations (GET, PUT, DELETE)
would be useful
- Counting the number of accesses to each region to detect hotspotting
- Exposing the current number of HLog files
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5533) Add more metrics to HBase
[ https://issues.apache.org/jira/browse/HBASE-5533?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13239129#comment-13239129 ] Lars Hofhansl commented on HBASE-5533: -- Now, do we fix it, or do I take it out of 0.94.0? Add more metrics to HBase - Key: HBASE-5533 URL: https://issues.apache.org/jira/browse/HBASE-5533 Project: HBase Issue Type: Improvement Affects Versions: 0.92.2, 0.94.0 Reporter: Shaneal Manek Assignee: Shaneal Manek Priority: Minor Fix For: 0.94.0, 0.96.0 Attachments: BlockingQueueContention.java, HBASE-5533-0.92-v4.patch, HBASE-5533-TRUNK-v6.patch, HBASE-5533-TRUNK-v6.patch, TimingOverhead.java, hbase-5533-0.92.patch, hbase5533-0.92-v2.patch, hbase5533-0.92-v3.patch, hbase5533-0.92-v5.patch, histogram_web_ui.png To debug/monitor production clusters, there are some more metrics I wish I had available. In particular: - Although the average FS latencies are useful, a 'histogram' of recent latencies (90% of reads completed in under 100ms, 99% in under 200ms, etc) would be more useful - Similar histograms of latencies on common operations (GET, PUT, DELETE) would be useful - Counting the number of accesses to each region to detect hotspotting - Exposing the current number of HLog files -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5533) Add more metrics to HBase
[ https://issues.apache.org/jira/browse/HBASE-5533?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13237406#comment-13237406 ] Shaneal Manek commented on HBASE-5533: -- Ah, upon further examination, I realized I initially misunderstood what that those HRegion metrics you pointed out were doing. They are calculating statistics on a per-CF basis (HBASE-4219), while I thought they were per-RS. Unfortunately, that means they don't overlap like I thought they did. I've attached a new patch that applies to TRUNK. Add more metrics to HBase - Key: HBASE-5533 URL: https://issues.apache.org/jira/browse/HBASE-5533 Project: HBase Issue Type: Improvement Affects Versions: 0.92.2, 0.94.0 Reporter: Shaneal Manek Assignee: Shaneal Manek Priority: Minor Attachments: BlockingQueueContention.java, HBASE-5533-0.92-v4.patch, TimingOverhead.java, hbase-5533-0.92.patch, hbase5533-0.92-v2.patch, hbase5533-0.92-v3.patch, hbase5533-0.92-v5.patch, histogram_web_ui.png To debug/monitor production clusters, there are some more metrics I wish I had available. In particular: - Although the average FS latencies are useful, a 'histogram' of recent latencies (90% of reads completed in under 100ms, 99% in under 200ms, etc) would be more useful - Similar histograms of latencies on common operations (GET, PUT, DELETE) would be useful - Counting the number of accesses to each region to detect hotspotting - Exposing the current number of HLog files -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5533) Add more metrics to HBase
[ https://issues.apache.org/jira/browse/HBASE-5533?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13236158#comment-13236158 ] Shaneal Manek commented on HBASE-5533: -- AFAIK, it's simply a bug to ever use System.currentTimeMillis() for timing things. Its output will jump around wildly with NTP updates, leap seconds, daylight savings time, timezone changes, and many other reasons (see: https://blogs.oracle.com/dholmes/entry/inside_the_hotspot_vm_clocks). I've been bitten by DST causing a ~10ms op to look like it took ~1hr. Which spiked our mean time high enough to set off an alerting system (another reason I like 'median' so much better than 'mean' too ;-)). System.nanotime() doesn't have these problems. There is definitely overlap between the new and old metrics (the old ones are, in fact, an exact subset of the new ones, since the new 'histogram' keeps track of the mean too). I didn't want to remove the old ones because it would have broken backwards compatibility with existing monitoring/alerting tools. Is there a method for marking metrics deprecated? I'm in progress on porting my patch to TRUNK, and consolidating measurements for both sets of metrics. I'll be uploading it later tonight - but just wanted to make a status update. Add more metrics to HBase - Key: HBASE-5533 URL: https://issues.apache.org/jira/browse/HBASE-5533 Project: HBase Issue Type: Improvement Affects Versions: 0.92.2, 0.94.0 Reporter: Shaneal Manek Assignee: Shaneal Manek Priority: Minor Attachments: BlockingQueueContention.java, HBASE-5533-0.92-v4.patch, TimingOverhead.java, hbase-5533-0.92.patch, hbase5533-0.92-v2.patch, hbase5533-0.92-v3.patch, hbase5533-0.92-v5.patch, histogram_web_ui.png To debug/monitor production clusters, there are some more metrics I wish I had available. In particular: - Although the average FS latencies are useful, a 'histogram' of recent latencies (90% of reads completed in under 100ms, 99% in under 200ms, etc) would be more useful - Similar histograms of latencies on common operations (GET, PUT, DELETE) would be useful - Counting the number of accesses to each region to detect hotspotting - Exposing the current number of HLog files -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5533) Add more metrics to HBase
[ https://issues.apache.org/jira/browse/HBASE-5533?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13236358#comment-13236358 ] stack commented on HBASE-5533: -- The above is all good. I'm not sure how to deprecate a metric. I've not done that before. We could just turn them off or emit the new stuff on the old metric name too (which seems silly). If the new are superior, we should just replace the old w/ the new? Would that be hard to do Shaneal? Add more metrics to HBase - Key: HBASE-5533 URL: https://issues.apache.org/jira/browse/HBASE-5533 Project: HBase Issue Type: Improvement Affects Versions: 0.92.2, 0.94.0 Reporter: Shaneal Manek Assignee: Shaneal Manek Priority: Minor Attachments: BlockingQueueContention.java, HBASE-5533-0.92-v4.patch, TimingOverhead.java, hbase-5533-0.92.patch, hbase5533-0.92-v2.patch, hbase5533-0.92-v3.patch, hbase5533-0.92-v5.patch, histogram_web_ui.png To debug/monitor production clusters, there are some more metrics I wish I had available. In particular: - Although the average FS latencies are useful, a 'histogram' of recent latencies (90% of reads completed in under 100ms, 99% in under 200ms, etc) would be more useful - Similar histograms of latencies on common operations (GET, PUT, DELETE) would be useful - Counting the number of accesses to each region to detect hotspotting - Exposing the current number of HLog files -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5533) Add more metrics to HBase
[ https://issues.apache.org/jira/browse/HBASE-5533?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13236357#comment-13236357 ] stack commented on HBASE-5533: -- The above is all good. I'm not sure how to deprecate a metric. I've not done that before. We could just turn them off or emit the new stuff on the old metric name too (which seems silly). If the new are superior, we should just replace the old w/ the new? Would that be hard to do Shaneal? Add more metrics to HBase - Key: HBASE-5533 URL: https://issues.apache.org/jira/browse/HBASE-5533 Project: HBase Issue Type: Improvement Affects Versions: 0.92.2, 0.94.0 Reporter: Shaneal Manek Assignee: Shaneal Manek Priority: Minor Attachments: BlockingQueueContention.java, HBASE-5533-0.92-v4.patch, TimingOverhead.java, hbase-5533-0.92.patch, hbase5533-0.92-v2.patch, hbase5533-0.92-v3.patch, hbase5533-0.92-v5.patch, histogram_web_ui.png To debug/monitor production clusters, there are some more metrics I wish I had available. In particular: - Although the average FS latencies are useful, a 'histogram' of recent latencies (90% of reads completed in under 100ms, 99% in under 200ms, etc) would be more useful - Similar histograms of latencies on common operations (GET, PUT, DELETE) would be useful - Counting the number of accesses to each region to detect hotspotting - Exposing the current number of HLog files -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5533) Add more metrics to HBase
[
https://issues.apache.org/jira/browse/HBASE-5533?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13234121#comment-13234121
]
stack commented on HBASE-5533:
--
I'm +1 on committing to trunk. Would like to see how it does there first
before bringing it back. This is a nice addition.
Shaneal, do you think there'll be issue because you are measuring here using
nanotime whereas elsewhere we do System.currentTimeMillis doing measurements?
Do you think this will ever make for discrepency or do you think that in the
wash, a slow call is a slow call and between the jigs and reels, the nanotime
count will be about same as millisecond counts?
Also, what about these calls down in HRegion.get and deep in put where we do:
{code}
long now = EnvironmentEdgeManager.currentTimeMillis();
{code}
I see the now is used to do things like:
{code}
HRegion.incrTimeVaryingMetric(metricPrefix + put_, after - now);
{code}
... is there overlap between these old metrics and what you are adding? Or,
since your additions work at a higher level up in HRegionServer rather long
after the get or put has started, should we be pulling these metrics up into
HRS so they envelope more of the server operation than they currently do?
Good stuff.
Add more metrics to HBase
-
Key: HBASE-5533
URL: https://issues.apache.org/jira/browse/HBASE-5533
Project: HBase
Issue Type: Improvement
Affects Versions: 0.92.2, 0.94.0
Reporter: Shaneal Manek
Assignee: Shaneal Manek
Priority: Minor
Attachments: BlockingQueueContention.java, HBASE-5533-0.92-v4.patch,
TimingOverhead.java, hbase-5533-0.92.patch, hbase5533-0.92-v2.patch,
hbase5533-0.92-v3.patch, hbase5533-0.92-v5.patch, histogram_web_ui.png
To debug/monitor production clusters, there are some more metrics I wish I
had available.
In particular:
- Although the average FS latencies are useful, a 'histogram' of recent
latencies (90% of reads completed in under 100ms, 99% in under 200ms, etc)
would be more useful
- Similar histograms of latencies on common operations (GET, PUT, DELETE)
would be useful
- Counting the number of accesses to each region to detect hotspotting
- Exposing the current number of HLog files
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5533) Add more metrics to HBase
[ https://issues.apache.org/jira/browse/HBASE-5533?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13231733#comment-13231733 ] stack commented on HBASE-5533: -- Those metrics look beautiful. NamedThreadFactory overlaps with functionality in the (badly named) util.Threads class. You might check it out to make sure we're not duplicating facility. No need of these' + * Copyright 2010 The Apache Software Foundation' lines when you are adding headers. TestMetricsHistogram is missing license header. Use something like http://hadoop.apache.org/common/docs/current/api/org/apache/hadoop/util/StringUtils.html#limitDecimalTo2%28double%29 when you are in RegionServerMetrics#toString outputting your new values else they could go ugly with lots of decimal places (see your screen shot). Fix the minors above and lets get it in. Good stuff Shaneal. Add more metrics to HBase - Key: HBASE-5533 URL: https://issues.apache.org/jira/browse/HBASE-5533 Project: HBase Issue Type: Improvement Affects Versions: 0.92.2, 0.94.0 Reporter: Shaneal Manek Assignee: Shaneal Manek Priority: Minor Attachments: BlockingQueueContention.java, HBASE-5533-0.92-v4.patch, TimingOverhead.java, hbase-5533-0.92.patch, hbase5533-0.92-v2.patch, hbase5533-0.92-v3.patch, histogram_web_ui.png To debub/monitor production clusters, there are some more metrics I wish I had available. In particular: - Although the average FS latencies are useful, a 'histogram' of recent latencies (90% of reads completed in under 100ms, 99% in under 200ms, etc) would be more useful - Similar histograms of latencies on common operations (GET, PUT, DELETE) would be useful - Counting the number of accesses to each region to detect hotspotting - Exposing the current number of HLog files -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5533) Add more metrics to HBase
[ https://issues.apache.org/jira/browse/HBASE-5533?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13230792#comment-13230792 ] Shaneal Manek commented on HBASE-5533: -- Re line length: Ah, sorry about that. I was using 120 - fixed and will use 80 in the future. Assign and declare together: Changed there (and in several other places). Will assign with declaration where possible in the future. Re timing overhead: Fixed it to cache calls to System.currentTimeMillis where possible (see: CURRENT_TICK). Unfortunately, there is no good way to convert between System.nanoTime and System.currentTimeMillis since the nanoTime epoch is undefined. I've attached a benchmark showing that the overhead for System.nanoTime() is ~20ns. At the rate we're calling it that should be negligible. Load: I couldn't measure a difference in performance with this code and without it. I've used a similar technique to get latency distributions on a service handling 10K requests/second with no trouble, fwiw. TSDB/UI: Yep, TSDB should be able to interpret it as a standard metric (so you can see you 95th percentile latencies over time, for example). I've uploaded a screenshot so you can see what it looks like in the UI. Add more metrics to HBase - Key: HBASE-5533 URL: https://issues.apache.org/jira/browse/HBASE-5533 Project: HBase Issue Type: Improvement Affects Versions: 0.92.2, 0.94.0 Reporter: Shaneal Manek Assignee: Shaneal Manek Priority: Minor Attachments: BlockingQueueContention.java, HBASE-5533-0.92-v4.patch, hbase-5533-0.92.patch, hbase5533-0.92-v2.patch, hbase5533-0.92-v3.patch, histogram_web_ui.png To debub/monitor production clusters, there are some more metrics I wish I had available. In particular: - Although the average FS latencies are useful, a 'histogram' of recent latencies (90% of reads completed in under 100ms, 99% in under 200ms, etc) would be more useful - Similar histograms of latencies on common operations (GET, PUT, DELETE) would be useful - Counting the number of accesses to each region to detect hotspotting - Exposing the current number of HLog files -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5533) Add more metrics to HBase
[
https://issues.apache.org/jira/browse/HBASE-5533?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13226238#comment-13226238
]
stack commented on HBASE-5533:
--
Line lengths are usually 80 chars in our src code Shaneal.
Minor, assign and declare on the one line the below rather than wait till
constructor?
{code}
+this.counts = new MapMaker().makeComputingMap(new FunctionString,
Counter() {
+ @Override
+ public Counter apply(String input) {
+return new Counter();
+ }
+});
{code}
This looks cool... ExponentiallyDecayingSample
Snapshot looks excellent.
UniformSample looks sweet
On this...
{code}
+final long startTime = System.nanoTime();
{code}
... aren't we getting a currentTimeMillis soon after? Do we want to be doing
all this time getting? We should minimize as much as we can? Check it out I'd
say. Also, there is EdgeEnvironment or EnvironmentEdge that we use for system
things like getting time because we want to have a layer between us and system
especially when testing ... so we can mess things up. You might want to check
it out.
Tests look good.
How does this stuff look for a server under load? Will say tsdb be able to
make sense of it? Hows it look in the ui? I suppose I could just apply the
patch and try it but you might have some pictures laying around.
How did we make it this far w/o this stuff?
Add more metrics to HBase
-
Key: HBASE-5533
URL: https://issues.apache.org/jira/browse/HBASE-5533
Project: HBase
Issue Type: Improvement
Affects Versions: 0.92.0
Reporter: Shaneal Manek
Assignee: Shaneal Manek
Priority: Minor
Attachments: BlockingQueueContention.java, hbase-5533-0.92.patch,
hbase5533-0.92-v2.patch, hbase5533-0.92-v3.patch
To debub/monitor production clusters, there are some more metrics I wish I
had available.
In particular:
- Although the average FS latencies are useful, a 'histogram' of recent
latencies (90% of reads completed in under 100ms, 99% in under 200ms, etc)
would be more useful
- Similar histograms of latencies on common operations (GET, PUT, DELETE)
would be useful
- Counting the number of accesses to each region to detect hotspotting
- Exposing the current number of HLog files
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5533) Add more metrics to HBase
[ https://issues.apache.org/jira/browse/HBASE-5533?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13223516#comment-13223516 ] Hadoop QA commented on HBASE-5533: -- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12517268/hbase-5533-0.92.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 4 new or modified tests. -1 patch. The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/1116//console This message is automatically generated. Add more metrics to HBase - Key: HBASE-5533 URL: https://issues.apache.org/jira/browse/HBASE-5533 Project: HBase Issue Type: Improvement Affects Versions: 0.92.0 Reporter: Shaneal Manek Assignee: Shaneal Manek Priority: Minor Attachments: hbase-5533-0.92.patch To debub/monitor production clusters, there are some more metrics I wish I had available. In particular: - Although the average FS latencies are useful, a 'histogram' of recent latencies (90% of reads completed in under 100ms, 99% in under 200ms, etc) would be more useful - Similar histograms of latencies on common operations (GET, PUT, DELETE) would be useful - Counting the number of accesses to each region to detect hotspotting - Exposing the current number of HLog files -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5533) Add more metrics to HBase
[
https://issues.apache.org/jira/browse/HBASE-5533?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13223523#comment-13223523
]
stack commented on HBASE-5533:
--
Did you mean the below:
{code}
-# hbase.class=org.apache.hadoop.hbase.metrics.file.TimeStampingFileContext
-# hbase.period=10
-# hbase.fileName=/tmp/metrics_hbase.log
+hbase.class=org.apache.hadoop.hbase.metrics.file.TimeStampingFileContext
+hbase.period=10
+hbase.fileName=/tmp/metrics_hbase.log
{code}
Will there be a bunch of contention on these additions:
{code}
+ static volatile BlockingQueueLong fsReadLatenciesNanos = new
ArrayBlockingQueueLong(LATENCY_BUFFER_SIZE);
{code}
Could this fill the logs with thousands of repeated messages:
{code}
+ if (!stored) {
+LOG.warn(Dropping fs latency stat since buffer is full);
+ }
{code}
Could we use the cliff click counters instead of AtomicLong? They are on the
classpath IIRC:
{code}
+ private final MapString, AtomicLong counts;
{code}
These additions would be great to have.
Add more metrics to HBase
-
Key: HBASE-5533
URL: https://issues.apache.org/jira/browse/HBASE-5533
Project: HBase
Issue Type: Improvement
Affects Versions: 0.92.0
Reporter: Shaneal Manek
Assignee: Shaneal Manek
Priority: Minor
Attachments: hbase-5533-0.92.patch
To debub/monitor production clusters, there are some more metrics I wish I
had available.
In particular:
- Although the average FS latencies are useful, a 'histogram' of recent
latencies (90% of reads completed in under 100ms, 99% in under 200ms, etc)
would be more useful
- Similar histograms of latencies on common operations (GET, PUT, DELETE)
would be useful
- Counting the number of accesses to each region to detect hotspotting
- Exposing the current number of HLog files
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5533) Add more metrics to HBase
[ https://issues.apache.org/jira/browse/HBASE-5533?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13223727#comment-13223727 ] Hadoop QA commented on HBASE-5533: -- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12517316/BlockingQueueContention.java against trunk revision . +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. -1 patch. The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/1120//console This message is automatically generated. Add more metrics to HBase - Key: HBASE-5533 URL: https://issues.apache.org/jira/browse/HBASE-5533 Project: HBase Issue Type: Improvement Affects Versions: 0.92.0 Reporter: Shaneal Manek Assignee: Shaneal Manek Priority: Minor Attachments: BlockingQueueContention.java, hbase-5533-0.92.patch, hbase5533-0.92-v2.patch To debub/monitor production clusters, there are some more metrics I wish I had available. In particular: - Although the average FS latencies are useful, a 'histogram' of recent latencies (90% of reads completed in under 100ms, 99% in under 200ms, etc) would be more useful - Similar histograms of latencies on common operations (GET, PUT, DELETE) would be useful - Counting the number of accesses to each region to detect hotspotting - Exposing the current number of HLog files -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5533) Add more metrics to HBase
[ https://issues.apache.org/jira/browse/HBASE-5533?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13223914#comment-13223914 ] Hadoop QA commented on HBASE-5533: -- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12517353/hbase5533-0.92-v3.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 4 new or modified tests. -1 patch. The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/1126//console This message is automatically generated. Add more metrics to HBase - Key: HBASE-5533 URL: https://issues.apache.org/jira/browse/HBASE-5533 Project: HBase Issue Type: Improvement Affects Versions: 0.92.0 Reporter: Shaneal Manek Assignee: Shaneal Manek Priority: Minor Attachments: BlockingQueueContention.java, hbase-5533-0.92.patch, hbase5533-0.92-v2.patch, hbase5533-0.92-v3.patch To debub/monitor production clusters, there are some more metrics I wish I had available. In particular: - Although the average FS latencies are useful, a 'histogram' of recent latencies (90% of reads completed in under 100ms, 99% in under 200ms, etc) would be more useful - Similar histograms of latencies on common operations (GET, PUT, DELETE) would be useful - Counting the number of accesses to each region to detect hotspotting - Exposing the current number of HLog files -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
