[jira] [Commented] (HBASE-20904) Prometheus /metrics http endpoint for monitoring integration
[
https://issues.apache.org/jira/browse/HBASE-20904?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17584629#comment-17584629
]
Hudson commented on HBASE-20904:
Results for branch branch-2
[build #626 on
builds.a.o|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-2/626/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-2/626/General_20Nightly_20Build_20Report/]
(/) {color:green}+1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-2/626/JDK8_20Nightly_20Build_20Report_20_28Hadoop2_29/]
(/) {color:green}+1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-2/626/JDK8_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(x) {color:red}-1 jdk11 hadoop3 checks{color}
-- For more information [see jdk11
report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-2/626/JDK11_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> Prometheus /metrics http endpoint for monitoring integration
>
>
> Key: HBASE-20904
> URL: https://issues.apache.org/jira/browse/HBASE-20904
> Project: HBase
> Issue Type: New Feature
> Components: metrics, monitoring
>Reporter: Hari Sekhon
>Assignee: Luca Kovacs
>Priority: Major
> Fix For: 2.6.0, 3.0.0-alpha-4
>
>
> Feature Request to add Prometheus /metrics http endpoint for monitoring
> integration:
> [https://prometheus.io/docs/prometheus/latest/configuration/configuration/#%3Cscrape_config%3E]
> Prometheus metrics format for that endpoint:
> [https://github.com/prometheus/docs/blob/master/content/docs/instrumenting/exposition_formats.md]
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (HBASE-20904) Prometheus /metrics http endpoint for monitoring integration
[
https://issues.apache.org/jira/browse/HBASE-20904?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17584524#comment-17584524
]
Hudson commented on HBASE-20904:
Results for branch master
[build #665 on
builds.a.o|https://ci-hbase.apache.org/job/HBase%20Nightly/job/master/665/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/master/665/General_20Nightly_20Build_20Report/]
(/) {color:green}+1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/master/665/JDK8_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(x) {color:red}-1 jdk11 hadoop3 checks{color}
-- For more information [see jdk11
report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/master/665/JDK11_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> Prometheus /metrics http endpoint for monitoring integration
>
>
> Key: HBASE-20904
> URL: https://issues.apache.org/jira/browse/HBASE-20904
> Project: HBase
> Issue Type: New Feature
> Components: metrics, monitoring
>Reporter: Hari Sekhon
>Assignee: Luca Kovacs
>Priority: Major
> Fix For: 2.6.0, 3.0.0-alpha-4
>
>
> Feature Request to add Prometheus /metrics http endpoint for monitoring
> integration:
> [https://prometheus.io/docs/prometheus/latest/configuration/configuration/#%3Cscrape_config%3E]
> Prometheus metrics format for that endpoint:
> [https://github.com/prometheus/docs/blob/master/content/docs/instrumenting/exposition_formats.md]
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (HBASE-20904) Prometheus /metrics http endpoint for monitoring integration
[ https://issues.apache.org/jira/browse/HBASE-20904?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17583971#comment-17583971 ] Duo Zhang commented on HBASE-20904: --- [~lkovacs] Please remember to fill the release note as this is a new feature. Thanks. > Prometheus /metrics http endpoint for monitoring integration > > > Key: HBASE-20904 > URL: https://issues.apache.org/jira/browse/HBASE-20904 > Project: HBase > Issue Type: New Feature > Components: metrics, monitoring >Reporter: Hari Sekhon >Assignee: Luca Kovacs >Priority: Major > Fix For: 2.6.0, 3.0.0-alpha-4 > > > Feature Request to add Prometheus /metrics http endpoint for monitoring > integration: > [https://prometheus.io/docs/prometheus/latest/configuration/configuration/#%3Cscrape_config%3E] > Prometheus metrics format for that endpoint: > [https://github.com/prometheus/docs/blob/master/content/docs/instrumenting/exposition_formats.md] > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (HBASE-20904) Prometheus /metrics http endpoint for monitoring integration
[ https://issues.apache.org/jira/browse/HBASE-20904?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17575172#comment-17575172 ] Luca Kovacs commented on HBASE-20904: - Dear [~mmpataki], I would like continue this project. Do you have any future plans with it or are you against it? Thanks in advance :) > Prometheus /metrics http endpoint for monitoring integration > > > Key: HBASE-20904 > URL: https://issues.apache.org/jira/browse/HBASE-20904 > Project: HBase > Issue Type: New Feature > Components: metrics, monitoring >Reporter: Hari Sekhon >Assignee: Madhusoodan >Priority: Major > > Feature Request to add Prometheus /metrics http endpoint for monitoring > integration: > [https://prometheus.io/docs/prometheus/latest/configuration/configuration/#%3Cscrape_config%3E] > Prometheus metrics format for that endpoint: > [https://github.com/prometheus/docs/blob/master/content/docs/instrumenting/exposition_formats.md] > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (HBASE-20904) Prometheus /metrics http endpoint for monitoring integration
[ https://issues.apache.org/jira/browse/HBASE-20904?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17160287#comment-17160287 ] Nick Dimiduk commented on HBASE-20904: -- I'm not keen on copying more metrics code from Hadoop in to HBase. I'm also not keen on expanding our codebase when perfectly good alternatives exist that are supported by the project with which we're trying to integrate. I'm not saying -0, I'm just asking for strong justification. It looks like there's someone [actively working|https://github.com/prometheus/jmx_exporter/pull/459] on making Prometheus/HBase integration work better. Why not team up with them to make the agent more robust for our users? Also, isn't the Slider project [retired|https://incubator.apache.org/projects/slider.html]? bq. we have to dynamically decide the port on which agent listens on as two server processes (M / RS) can be deployed on a single host (by slider/yarn). This becomes a burden for the slider user as agent doesn't provide this funcationality and one cannot choose some random number <64k. Having the agent support binding to an arbitrary port and advertising it back to an operator seems like a feature the agent should have, file a feature request? I haven't used slider so I don't know what limitations are present on that platform, but any host running multiple JVMs with this agent installed will have a similar need, so seems like an upstream problem they want to solve anyway. bq. Well we can use agent+pushgateways combination but the metrics are published all the time which may be become unnecessary given the size of metrics published (prometheus expo format is verbose). I don't quite follow you here... I see the agent supports allow/deny lists for metrics, running as an agent or the scraping gateway. Does this not work to limit the data transferred? To implement this as a servlet, we would be generating all the metrics (as we do now, with JSON), which ends up transferring all the data as well. bq. these dynamically chosen ports have to be (discovered by) / (told to) prometheus scraper. This is again a burden for the slider user as the HBase can be run as a Again, I'm not sure about Slider, but service discovery seems like a problem that Slider needs to solve in general, so some interface between that mechanism in Slider and the dynamic port binding improvement in the agent would be required. bq. Some customers are skeptical about opening new ports for an agent. I believe the intended deployment model of the agent to bind to localhost. A prometheus scraper agent, also running on localhost, can query the data. In that case, there's no opening a port to the network, just a local socket between processes. > Prometheus /metrics http endpoint for monitoring integration > > > Key: HBASE-20904 > URL: https://issues.apache.org/jira/browse/HBASE-20904 > Project: HBase > Issue Type: New Feature > Components: metrics, monitoring >Reporter: Hari Sekhon >Assignee: Madhusoodan >Priority: Major > > Feature Request to add Prometheus /metrics http endpoint for monitoring > integration: > [https://prometheus.io/docs/prometheus/latest/configuration/configuration/#%3Cscrape_config%3E] > Prometheus metrics format for that endpoint: > [https://github.com/prometheus/docs/blob/master/content/docs/instrumenting/exposition_formats.md] > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-20904) Prometheus /metrics http endpoint for monitoring integration
[ https://issues.apache.org/jira/browse/HBASE-20904?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17159718#comment-17159718 ] Madhusoodan commented on HBASE-20904: - [~ndimiduk] Here are the pros and cons of using the agent, please comment your views. Pros * code management is no more a HBase developer burden Cons * On slider based deployments.. ** we have to dynamically decide the port on which agent listens on as two server processes (M / RS) can be deployed on a single host (by slider/yarn). This becomes a burden for the slider user as agent doesn't provide this funcationality and one cannot choose some random number <64k. Well we can use agent+pushgateways combination but the metrics are published all the time which may be become unnecessary given the size of metrics published (prometheus expo format is verbose). ** these dynamically chosen ports have to be (discovered by) / (told to) prometheus scraper. This is again a burden for the slider user as the HBase can be run as a * Some customers are skeptical about opening new ports for an agent. > Prometheus /metrics http endpoint for monitoring integration > > > Key: HBASE-20904 > URL: https://issues.apache.org/jira/browse/HBASE-20904 > Project: HBase > Issue Type: New Feature > Components: metrics, monitoring >Reporter: Hari Sekhon >Assignee: Madhusoodan >Priority: Major > > Feature Request to add Prometheus /metrics http endpoint for monitoring > integration: > [https://prometheus.io/docs/prometheus/latest/configuration/configuration/#%3Cscrape_config%3E] > Prometheus metrics format for that endpoint: > [https://github.com/prometheus/docs/blob/master/content/docs/instrumenting/exposition_formats.md] > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-20904) Prometheus /metrics http endpoint for monitoring integration
[ https://issues.apache.org/jira/browse/HBASE-20904?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17158524#comment-17158524 ] Nick Dimiduk commented on HBASE-20904: -- Apologies for arriving late to the discussion, it appears a fair bit of development has already happened. However, I'm concerned -- why would we build this ourselves when there exists [a tool provided by the prometheus project|https://github.com/prometheus/jmx_exporter] for exporting JMX beans? I have not tried it, but I have had very good success using [a tool for exporting JMX beans over http/json|https://jolokia.org] that appears to function identically. Why not avoid writing all the code and instead document use of the existing agent? > Prometheus /metrics http endpoint for monitoring integration > > > Key: HBASE-20904 > URL: https://issues.apache.org/jira/browse/HBASE-20904 > Project: HBase > Issue Type: New Feature > Components: metrics, monitoring >Reporter: Hari Sekhon >Assignee: Madhusoodan >Priority: Major > > Feature Request to add Prometheus /metrics http endpoint for monitoring > integration: > [https://prometheus.io/docs/prometheus/latest/configuration/configuration/#%3Cscrape_config%3E] > Prometheus metrics format for that endpoint: > [https://github.com/prometheus/docs/blob/master/content/docs/instrumenting/exposition_formats.md] > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-20904) Prometheus /metrics http endpoint for monitoring integration
[ https://issues.apache.org/jira/browse/HBASE-20904?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17126359#comment-17126359 ] Sean Busbey commented on HBASE-20904: - @mmpataki I've added you to the contributor group in jira so you should be able to assign this to yourself now. Please place it in "patch available" status since you already have made a PR. > Prometheus /metrics http endpoint for monitoring integration > > > Key: HBASE-20904 > URL: https://issues.apache.org/jira/browse/HBASE-20904 > Project: HBase > Issue Type: New Feature > Components: metrics, monitoring >Reporter: Hari Sekhon >Priority: Major > > Feature Request to add Prometheus /metrics http endpoint for monitoring > integration: > [https://prometheus.io/docs/prometheus/latest/configuration/configuration/#%3Cscrape_config%3E] > Prometheus metrics format for that endpoint: > [https://github.com/prometheus/docs/blob/master/content/docs/instrumenting/exposition_formats.md] > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-20904) Prometheus /metrics http endpoint for monitoring integration
[ https://issues.apache.org/jira/browse/HBASE-20904?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17121216#comment-17121216 ] Michael Stack commented on HBASE-20904: --- [~mmpataki] added helpful note up on the PR. I'm adding it here too. bq. HBASE-9774 brought in hbase-native metrics collection for the coprocessors and a decision was made to to use this API to record all the other metrics as well (hbase-metrics-api / README.txt). I am referring to this as "new API'. Now all the work to consume this new API isn't done so we still have to depend on the "old", "hadoop2" based metric collection. > Prometheus /metrics http endpoint for monitoring integration > > > Key: HBASE-20904 > URL: https://issues.apache.org/jira/browse/HBASE-20904 > Project: HBase > Issue Type: New Feature > Components: metrics, monitoring >Reporter: Hari Sekhon >Priority: Major > > Feature Request to add Prometheus /metrics http endpoint for monitoring > integration: > [https://prometheus.io/docs/prometheus/latest/configuration/configuration/#%3Cscrape_config%3E] > Prometheus metrics format for that endpoint: > [https://github.com/prometheus/docs/blob/master/content/docs/instrumenting/exposition_formats.md] > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-20904) Prometheus /metrics http endpoint for monitoring integration
[ https://issues.apache.org/jira/browse/HBASE-20904?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17120482#comment-17120482 ] Madhusoodan commented on HBASE-20904: - As you may already know we built a new API to record metrics, I am referring to it as new API. (not sure, when it started but referring it as "new" :P) > Prometheus /metrics http endpoint for monitoring integration > > > Key: HBASE-20904 > URL: https://issues.apache.org/jira/browse/HBASE-20904 > Project: HBase > Issue Type: New Feature > Components: metrics, monitoring >Reporter: Hari Sekhon >Priority: Major > > Feature Request to add Prometheus /metrics http endpoint for monitoring > integration: > [https://prometheus.io/docs/prometheus/latest/configuration/configuration/#%3Cscrape_config%3E] > Prometheus metrics format for that endpoint: > [https://github.com/prometheus/docs/blob/master/content/docs/instrumenting/exposition_formats.md] > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-20904) Prometheus /metrics http endpoint for monitoring integration
[ https://issues.apache.org/jira/browse/HBASE-20904?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17120405#comment-17120405 ] Michael Stack commented on HBASE-20904: --- What is the 'new metrics API'? Thanks. Left a question on the PR. > Prometheus /metrics http endpoint for monitoring integration > > > Key: HBASE-20904 > URL: https://issues.apache.org/jira/browse/HBASE-20904 > Project: HBase > Issue Type: New Feature > Components: metrics, monitoring >Reporter: Hari Sekhon >Priority: Major > > Feature Request to add Prometheus /metrics http endpoint for monitoring > integration: > [https://prometheus.io/docs/prometheus/latest/configuration/configuration/#%3Cscrape_config%3E] > Prometheus metrics format for that endpoint: > [https://github.com/prometheus/docs/blob/master/content/docs/instrumenting/exposition_formats.md] > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-20904) Prometheus /metrics http endpoint for monitoring integration
[ https://issues.apache.org/jira/browse/HBASE-20904?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17120330#comment-17120330 ] Madhusoodan commented on HBASE-20904: - [~busbey] can you please assign this to me > Prometheus /metrics http endpoint for monitoring integration > > > Key: HBASE-20904 > URL: https://issues.apache.org/jira/browse/HBASE-20904 > Project: HBase > Issue Type: New Feature > Components: metrics, monitoring >Reporter: Hari Sekhon >Priority: Major > > Feature Request to add Prometheus /metrics http endpoint for monitoring > integration: > [https://prometheus.io/docs/prometheus/latest/configuration/configuration/#%3Cscrape_config%3E] > Prometheus metrics format for that endpoint: > [https://github.com/prometheus/docs/blob/master/content/docs/instrumenting/exposition_formats.md] > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-20904) Prometheus /metrics http endpoint for monitoring integration
[ https://issues.apache.org/jira/browse/HBASE-20904?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17120316#comment-17120316 ] Madhusoodan commented on HBASE-20904: - I have worked on this today and submitted a PR (1814). *Implementation* Since all the metricsources have not moved to new metrics API, I added two servlets. a. /prom (PrometheusServlet) uses the new metrics API. b. /prom2 (PrometheusHadoop2Servlet) uses the old hadoop2 metrics API PrometheusHadoop2Servlet can be thrown out once all the metricsources start using the new API. Side note # Most of the servlet code for /prom2 code is cloned from HADOOP-16398 > Prometheus /metrics http endpoint for monitoring integration > > > Key: HBASE-20904 > URL: https://issues.apache.org/jira/browse/HBASE-20904 > Project: HBase > Issue Type: New Feature > Components: metrics, monitoring >Reporter: Hari Sekhon >Priority: Major > > Feature Request to add Prometheus /metrics http endpoint for monitoring > integration: > [https://prometheus.io/docs/prometheus/latest/configuration/configuration/#%3Cscrape_config%3E] > Prometheus metrics format for that endpoint: > [https://github.com/prometheus/docs/blob/master/content/docs/instrumenting/exposition_formats.md] > -- This message was sent by Atlassian Jira (v8.3.4#803005)
