Re: 1 minutes vs 5 minutes Metrics via Prometheus
Hi Now I have checked all metrics and the following metrics have values going from 0 up to X and back to 0 within 3 scrapings: nifi_amount_items_queued nifi_amount_threads_active nifi_percent_used_bytes nifi_percent_used_count nifi_size_content_queued_total nifi_transmitting The following metrics have been checked with flows with 1-2 files per 10 minutes. These shows the same values for 5 minutes however if there are many files they show changes at every scrap (15s). but still one file is effecting the value for 5 minutes. nifi_amount_bytes_read nifi_amount_bytes_received nifi_amount_bytes_sent nifi_amount_bytes_transferred nifi_amount_bytes_written nifi_amount_flowfiles_received nifi_amount_flowfiles_removed nifi_amount_flowfiles_sent nifi_amount_flowfiles_transferred nifi_amount_items_input nifi_amount_items_output nifi_average_lineage_duration nifi_size_content_input_total nifi_size_content_output_total nifi_total_bytes_read nifi_total_bytes_written kind regards Jens M. Kofoed -- Sent from: http://apache-nifi-users-list.2361937.n4.nabble.com/
Re: 1 minutes vs 5 minutes Metrics via Prometheus
Hi again, This is a bit more complex because I see some metrics change every time Prometheus is scraping (15 sec.) The following metrics is updated frequently: nifi_size_content_queued_total nifi_amount_items_queued nifi_amount_items_output nifi_amount_bytes_received But if the rate of flowfiles is very low (1 every 5-10 minutes) nifi_size_content_input_total is showing the same value for 5 minutes. The value change then new files arrived to the process (about every 1 minutes) but the values looks like it comes from the 5 minutes stat. Not like the metrics above. kind regards Jens M. Kofoed -- Sent from: http://apache-nifi-users-list.2361937.n4.nabble.com/
1 minutes vs 5 minutes Metrics via Prometheus
We are looking into getting metrics from NIFI into grafana via Prometheus. It seems the metrics reported are the 5 minutes stats and not the 1 minutes snap. If we look at the status history of a processor we can see that the history is shown in a 1 minutes interval. I know the stat on the processor is showing the last 5 minutes and it seems like it is the same 5 minutes stat that is being reported at the metrics instead of the 1 minutes snap. Is it possible to configure the Prometheus reporting task to report the 1 minutes snap instead? kind regards Jens M. Kofoed
Re: 5 minutes
Alessio, As Bryan mentioned, we do allow for Reporting Tasks to report these sorts of metrics and bulletins to other systems. One thing that I wanted to note, though, is that you can right-click on a Processor and view Status History. This will show you the metrics that the processor shows (and even more) over a much longer time window. It doesn't sum up the metrics, for instance, over a 1-hour period, but it shows the 5-minute stats over a long period. By default, NiFi is configured to take a snapshot of the 5-minute stats every 1 minute and hold the last 1 day worth of stats. This is configurable in nifi.properties. The only down side here is that these stats reside in memory and are lost on restart. It would be nice to have a mechanism for storing this information, but it's not something that's been done yet. Thanks -Mark On Jan 13, 2017, at 9:16 AM, Bryan Bende <bbe...@gmail.com<mailto:bbe...@gmail.com>> wrote: Hello, Generally NiFi is not meant to be a long term store of this type of information. There is an extension point called a ReportingTask which has access to the metrics seen in the UI, as well as the bulletin repository which are the error messages. You could implement a custom ReportingTask to push this information to whatever system you like to use for monitoring. The AmbariReportingTask would be a good example to look at, it pushes metrics to the Ambari metrics collector. Thanks, Bryan On Fri, Jan 13, 2017 at 6:21 AM, Alessio Palma <alessio.pa...@docomodigital.com<mailto:alessio.pa...@docomodigital.com>> wrote: Hello all, something into NIFI is timed to 5 minutes: processor statistic error messages How can I change processor statistics to hours and error messages to last days ?
Re: 5 minutes
Hello, Generally NiFi is not meant to be a long term store of this type of information. There is an extension point called a ReportingTask which has access to the metrics seen in the UI, as well as the bulletin repository which are the error messages. You could implement a custom ReportingTask to push this information to whatever system you like to use for monitoring. The AmbariReportingTask would be a good example to look at, it pushes metrics to the Ambari metrics collector. Thanks, Bryan On Fri, Jan 13, 2017 at 6:21 AM, Alessio Palma < alessio.pa...@docomodigital.com> wrote: > Hello all, > something into NIFI is timed to 5 minutes: > > processor statistic > error messages > > How can I change processor statistics to hours and error messages to last > days ? > >
5 minutes
Hello all, something into NIFI is timed to 5 minutes: processor statistic error messages How can I change processor statistics to hours and error messages to last days ?
