Hi guys,
I haven't specifically built functions to wrap this, but the data is easily
accessible via the API if that helps you.
Let us assume for a moment that a given flow has a Processor called
'PutSyslog' for which you wish to monitor the input and output queues. This
Processor receives flowfiles on a queue called 'success' and loops back
'failure, invalid' to itself for more processing.
Let us also assume that you have uniquely named your Processors. This can
be worked around if you haven't, it just makes the example easier.
In this example I have fed 4 flowfiles into the processor.
The following code uses NiPyApi to drive the NiFi Rest API, and retrieves
the upstream queue status for that processor on each call:
import nipyapi
proc_name = 'PutSyslog'
my_processor = nipyapi.canvas.get_processor(proc_name) # This gets the
object for the processor in question
pg_cons = nipyapi.canvas.get_connections(my_processor.status.group_id) #
this gets all connections in the parent process group to the processor
# if you are not familiar with Python, the following is a list
comprehension to produce a dict of queue_name:flowfiles_queued for each
connection listing our target processor as the destination.
[{con.status.name: con.status.aggregate_snapshot.queued} for con in
pg_cons.connections if proc_name in con.status.destination_name]
>> [{'success': '4 (0 bytes)'}, {'failure, invalid': '0 (0 bytes)'}]
Hope this helps,
Dan
On Thu, Mar 8, 2018 at 1:48 PM Abdou B <abour...@gmail.com> wrote:
> Hello guys,
>
> I am also interested in this discussion.
> I saw the blog post of Pierre :
>
> https://pierrevillard.com/2017/05/13/monitoring-nifi-site2site-reporting-tasks/
>
>
>
> Unfortunately , I have versions of Nifi older than 1.2.
> You may view/analyze Nifi built monitoring.
>
> If you go close to logout. Click on Summary.
> Go to the Connections Tab.
> You may sort by Queue / Size Thresold
>
>
> But i did not find any SiteToSiteStatusReportingTask...
> So I would like to know if there is any way to make some reporting on Nifi
> 1.1.0 ?
>
>
> Best Regards
>
> Abdou
>
> 2018-03-08 7:31 GMT+01:00 Pierre Villard <pierre.villard...@gmail.com>:
>
>> Hi Jon,
>>
>> You could use the S2S Status reporting task and filter the data on
>> Connection and, if need, on a specific connection name. It'll send you JSON
>> data containing the information about the queue (stats, size, backpressure,
>> etc).
>>
>> Pierre
>>
>> 2018-03-08 6:12 GMT+01:00 Jon Logan <jmlo...@buffalo.edu>:
>>
>>> Hi All,
>>>
>>> I am trying to get more specific queue sizes in a reporting task for
>>> metric reporting, but I can't seem to find out a way to do it -- the only
>>> methods exposed seem to be showing the total of all queue sizes. Is there a
>>> way that I am missing? I am trying to find a specific queue in front of one
>>> of my processors and grab that size.
>>>
>>> I also considered trying to do this via the Rest API but I couldn't
>>> quite find a way to get some of the IDs required to get some of the hops
>>> required through the API (namely, how to list components doesn't seem
>>> clear).
>>>
>>>
>>> Thanks in advance,
>>> Jon
>>>
>>
>>
>
