Noble, We have to expose those via the interface vs trying to cast to some expected object. For things like NIFI-8239 [1] we'll expose new properties from an object the rest of the code can't get to (and rightfully so), and I suspect the same would go for the properties of FlowController that you're looking to expose. If NIFI-8239 isn't capturing what you want, please feel free to add to that Jira or create a New Feature case to describe what you're looking for.
Regards, Matt [1] https://issues.apache.org/jira/browse/NIFI-8239 On Fri, Feb 12, 2021 at 2:35 AM Noble Numbat <[email protected]> wrote: > > Hi Matt > > Thanks for the clarification – I see your point. Both thread metrics > [1] are constant and at this stage we are trying to get them in the > right location. > > We have updated the appropriate Registry classes and > PrometheusMetricsUtil and the additional JVM metrics are displaying in > both the REST endpoint and the Reporting Task. The problem we > encountered was that the data for the metrics for the threads [1] and > the repositories [2] isn't available from the ReportingContext [3] > interface and we are reluctant to add methods to this interface as it > is also implemented by other classes that don’t contain the relevant > fields. > > The actual object that is passed to the PrometheusReportingTask’s > onTrigger method [4] is a StandardReportingContext [5] and this class > has the FlowController field that contains the thread and repository > information we need. I tried to cast the passed object to a > StandardReportingContext object so I could access the information but > got a java.lang.ClassCastException, which I believe is caused by > classes loaded with different classLoaders. The stack trace is below > [6]. I tried to resolve this by adding the file > org.apache.nifi.controller.reporting.ReportingContext under the > META-INF.services directory [7] to force the classLoader to load this > class. The file contained the line > org.apache.nifi.controller.reporting.StandardReportingContext. The > ClassCastException didn’t change. > > How would you suggest accessing the maxTimerDrivenThreads, > maxEventDrivenThreads, contentRepository, flowFileRepository and > provenanceRepository fields that are available via the FlowController > instance that is in StandardReportingContext when it is passed as a > ReportingContext? > > thanks > > [1] max_timer_driven_threads and max_event_driven_threads > [2] contentRepository, flowFileRepository and provenanceRepository > [3] > https://github.com/apache/nifi/blob/main/nifi-api/src/main/java/org/apache/nifi/reporting/ReportingContext.java > [4] > https://github.com/apache/nifi/blob/main/nifi-nar-bundles/nifi-prometheus-bundle/nifi-prometheus-reporting-task/src/main/java/org/apache/nifi/reporting/prometheus/PrometheusReportingTask.java#L196 > [5] > https://github.com/apache/nifi/blob/main/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/java/org/apache/nifi/controller/reporting/StandardReportingContext.java > [6] 2021-02-12 11:15:29,434 ERROR [Timer-Driven Process Thread-3] > o.a.n.r.p.PrometheusReportingTask > PrometheusReportingTask[id=938b946e-0177-1000-221f-68f6ba8694a6] : > java.lang.ClassCastException: > org.apache.nifi.controller.reporting.StandardReportingContext cannot > be cast to org.apache.nifi.controller.reporting.StandardReportingContext > java.lang.ClassCastException: > org.apache.nifi.controller.reporting.StandardReportingContext cannot > be cast to org.apache.nifi.controller.reporting.StandardReportingContext > at > org.apache.nifi.reporting.prometheus.PrometheusReportingTask.onTrigger(PrometheusReportingTask.java:296) > at > org.apache.nifi.controller.tasks.ReportingTaskWrapper.run(ReportingTaskWrapper.java:44) > at org.apache.nifi.engine.FlowEngine$2.run(FlowEngine.java:110) > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) > at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) > at > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) > at > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > at java.lang.Thread.run(Thread.java:748) > [7] > https://github.com/apache/nifi/blob/main/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/resources/META-INF/services > > > > > On Wed, 10 Feb 2021 at 06:42, Matt Burgess <[email protected]> wrote: > > > > The PrometheusReportingTask and the NIFi API endpoint for Prometheus > > metrics are different beasts but they use quite a bit of the same code > > [1]. The intent is to report on the same metrics wherever possible, > > and I think for the most part we've done that. They don't call each > > other, instead they get their own copies of the metrics registries, > > and they populate them when triggered. For the REST endpoint, it's > > done on-demand. For the Reporting Task, it's done when scheduled. The > > Reporting Task came first to provide a way for Prometheus to scrape a > > NiFi instance. But as Reporting Tasks are system-level controller > > services, they don't get exported to templates, possibly require > > configuration after manual instantiation, etc. To that end the REST > > endpoint was added, as it gets all the security, configuration, and > > HTTP server "for free" so to speak. Also I think the "totals" metrics > > might be for the whole cluster where the Reporting Task might only be > > for the node, but I'm not positive. > > > > For some of the metrics you added, aren't they constants based on > > properties or other settings? If so we probably didn't add them > > because it wasn't a useful metric on its own, but there is precedence > > for adding such static metrics for the purposes of downstream queries > > (used / max * 100% for example). > > > > The other ones (besides repository info) were possibly just oversights > > but if they are helpful metrics, then please feel free to add them. > > You should find that you can update the appropriate Registry classes > > as well as PrometheusMetricsUtil in nifi-prometheus-utils, and if no > > new registries are added, I believe both the REST endpoint and the > > Reporting Task will both have the new metrics. If you do need to add a > > registry (NiFiRepositoryMetricsRegistry for example), you'd want to > > follow the same pattern as the others and make the call to > > PrometheusMetricsUtil.createNiFiRepositoryMetrics() from both the > > endpoint [2] and the reporting task [3]. > > > > Last thing I'll mention is that we're using Dropwizard for most of > > these metrics, currently at version 4.1.2 but the latest 4.1.x release > > is 4.1.17. We might consider an upgrade while adding these metrics; > > not much has been done in the metrics-jvm module since 4.1.2 but a > > couple of new metrics were added [4], we could expose those as well. > > > > Regards, > > Matt > > > > [1] > > https://github.com/apache/nifi/tree/main/nifi-nar-bundles/nifi-extension-utils/nifi-prometheus-utils/src/main/java/org/apache/nifi/prometheus/util > > [2] > > https://github.com/apache/nifi/blob/main/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-api/src/main/java/org/apache/nifi/web/StandardNiFiServiceFacade.java#L5380 > > [3] > > https://github.com/apache/nifi/blob/main/nifi-nar-bundles/nifi-prometheus-bundle/nifi-prometheus-reporting-task/src/main/java/org/apache/nifi/reporting/prometheus/PrometheusReportingTask.java#L134 > > [4] > > https://github.com/dropwizard/metrics/commit/ccc91ef1ade1975d58595f23caa48d5ed68a6b54#diff-42e4dfff08e984191adc05ecf744f324f7a9039f72e26bafcb779876584e9e7b > > > > On Tue, Feb 9, 2021 at 2:49 AM Noble Numbat <[email protected]> > > wrote: > > > > > > Hi everyone, > > > > > > We have added metrics to the Prometheus metrics endpoint in the API > > > (/nifi-api/flow/metrics/prometheus) to improve programmatic access to > > > NiFi metrics for the purpose of monitoring. We’d like to contribute > > > these back to the project for the benefit of others. Please find the > > > list of metrics below. > > > > > > Before I open a JIRA ticket and pull request, I have some questions to > > > clarify my understanding and determine what else I will need to add to > > > the code. > > > 1. How are the use cases different between the Prometheus metrics > > > endpoint in the API (/nifi-api/flow/metrics/prometheus) and the > > > PrometheusReportingTask? I note that the metrics are almost identical > > > between the two. > > > 2. Is the intent to keep the metrics in these two endpoints the same? > > > That is, if we add metrics to the Prometheus metrics endpoint in the > > > API, are we expected to add these to the PrometheusReportingTask as > > > well? > > > 3. If so, one way to get the metrics data into > > > PrometheusReportingTask.java is to make an API call to > > > /nifi-api/controller/config. Is that an acceptable way to get metrics > > > data for max_event_driven_threads and max_timer_driven_threads? > > > > > > For context, here are the metrics we’ve added; > > > nifi_repository_max_bytes{flowfile} > > > nifi_repository_max_bytes{content} > > > nifi_repository_max_bytes{provenance} > > > nifi_repository_used_bytes{flowfile} > > > nifi_repository_used_bytes{content} > > > nifi_repository_used_bytes{provenance} > > > jvm_deadlocked_thread_count > > > max_event_driven_threads > > > max_timer_driven_threads > > > jvm_heap_non_init > > > jvm_heap_non_committed > > > jvm_heap_non_max > > > jvm_heap_non_used > > > jvm_heap_committed > > > jvm_heap_init > > > jvm_heap_max > > > > > > thanks
