gfexensa opened a new issue #6134: broker prometheus metrics export fails when bookie is launched by the broker URL: https://github.com/apache/pulsar/issues/6134 #### Expected behavior in our setting, the bookeeper was started at the same time as the broker with: % bin/pulsar broker --run-bookie --run-bookie-autorecovery and I was able to get bookkeeper metrics on 'http://pulsar-node-1.mydomain.net:8000/metrics' and I should had been able to get metrics from the broker too here : http://pulsar-node-1.mydomain.net:8080/metrics #### Actual behavior The following exception is raised at every request of the broker metrics. Jan 24 06:25:02 pulsar-node-1 pulsar[22776]: 06:25:02.139 [pulsar-web-59-8] INFO org.eclipse.jetty.server.RequestLog - 62.210.75.103 - - [24/Jan/2020:06:25:02 +0100] "GET /metrics HTTP/1.1" 302 0 "-" "Prometheus/2.1.0+ds" 0 Jan 24 06:25:02 pulsar-node-1 pulsar[22776]: 06:25:02.139 [prometheus-stats-68-1] ERROR org.apache.bookkeeper.common.util.SafeRunnable - Unexpected throwable caught Jan 24 06:25:02 pulsar-node-1 pulsar[22776]: java.lang.NoClassDefFoundError: Could not initialize class org.apache.pulsar.broker.stats.prometheus.PrometheusMetricsGenerator Jan 24 06:25:02 pulsar-node-1 pulsar[22776]: #011at org.apache.pulsar.broker.stats.prometheus.PrometheusMetricsServlet.lambda$doGet$0(PrometheusMetricsServlet.java:70) ~[org.apache.pulsar-pulsar-broker-2.4.2.jar:2.4.2] Jan 24 06:25:02 pulsar-node-1 pulsar[22776]: #011at org.apache.bookkeeper.mledger.util.SafeRun$1.safeRun(SafeRun.java:32) ~[org.apache.pulsar-managed-ledger-original-2.4.2.jar:2.4.2] Jan 24 06:25:02 pulsar-node-1 pulsar[22776]: #011at org.apache.bookkeeper.common.util.SafeRunnable.run(SafeRunnable.java:36) [org.apache.bookkeeper-bookkeeper-common-4.9.2.jar:4.9.2] Jan 24 06:25:02 pulsar-node-1 pulsar[22776]: #011at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_232] Jan 24 06:25:02 pulsar-node-1 pulsar[22776]: #011at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_232] Jan 24 06:25:02 pulsar-node-1 pulsar[22776]: #011at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) [?:1.8.0_232] Jan 24 06:25:02 pulsar-node-1 pulsar[22776]: #011at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) [?:1.8.0_232] Jan 24 06:25:02 pulsar-node-1 pulsar[22776]: #011at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_232] Jan 24 06:25:02 pulsar-node-1 pulsar[22776]: #011at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_232] Jan 24 06:25:02 pulsar-node-1 pulsar[22776]: #011at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) [io.netty-netty-all-4.1.32.Final.jar:4.1.32.Final] Jan 24 06:25:02 pulsar-node-1 pulsar[22776]: #011at java.lang.Thread.run(Thread.java:748) [?:1.8.0_232] Jan 24 06:25:02 pulsar-node-1 pulsar[22776]: 06:25:02.141 [pulsar-web-59-7] INFO org.eclipse.jetty.server.RequestLog - 62.210.75.103 - - [24/Jan/2020:06:24:32 +0100] "GET /metrics/ HTTP/1.1" 500 331 "http://pulsar-node-1.mydomain.net:8080/metrics"; "Prometheus/2.1.0+ds" 30000 Note that it's bookkeeper which is complaining that it can't found 'org.apache.pulsar.broker.stats.prometheus.PrometheusMetricsGenerator', which is a class from the broker. The 8080 port is the default port for the bookkeeper httpServerPort, so it's probably hiding the prometheus export. Unfortunatly, we can't change the prometheus port in the broker configuration. If both the bookie and the broker are launched separately, then both metrics are correctly exposed. #### Steps to reproduce Start the bookie along the broker and request the broker metric. #### System configuration **Pulsar version**: 2.4.2
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services
