I hadn't received Andy's response before I sent mine to the User's list. For
tracking, here is a link to my response:
http://mail-archives.apache.org/mod_mbox/nifi-users/201611.mbox/raw/%[email protected]%3E/2
Joe - - - - - - Joseph Percivalllinkedin.com/in/Percivalle:
[email protected]
On Wednesday, November 9, 2016 7:53 PM, Andy LoPresto
<[email protected]> wrote:
Sai,
I’d suggest you look at using a ControllerStatusReportingTask [1], which
monitors the processor and provides statistics from that component. If you need
to use this data within NiFi, you can also use
SiteToSiteProvenanceReportingTask [2], which can export provenance events as
data that can be consumed by (the same or a different) instance of NiFi. Both
of these may be overkill for your use case (the provenance reporting task will
offload all of the provenance events from the application), and if so, you may
be able to use counters [3] to do this quickly and easily (but be aware that
the values are just held in memory, so if you’re writing to ES hourly, you
should be ok, but they won’t persist across restart). Your initial thought to
use ExecuteScript would also work.
I believe Joe Percivall had done some work on SEP and window/aggregate
calculations before. That may also help with what you are doing.
Joe Percivall·4:51 PM
Here's the link to the processor:
https://github.com/JPercivall/nifi/blob/newRollingState/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/RollingWindowAggregator.java
Here's the ticket:
https://issues.apache.org/jira/browse/NIFI-1682?jql=project%20%3D%20NIFI%20AND%20text%20~%20%22rolling%20window%22
Keep in mind that this work is old and will need to be updated. I do have a
pending PR for UpdateAttribute with State though:
https://github.com/apache/nifi/pull/319
[1]
https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi.controller.ControllerStatusReportingTask/index.html[2]
https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi.reporting.SiteToSiteProvenanceReportingTask/index.html[3]
https://community.hortonworks.com/questions/50622/apache-nifi-what-are-counters-in-nifi.html
Andy [email protected]@gmail.comPGP Fingerprint:
70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69
On Nov 9, 2016, at 8:14 AM, Peddy, Sai <[email protected]> wrote:
Hi All,
I’m currently working on a use case to be able to track the number of
individual logs that come in and put that information in ElasticSearch. I
wanted to see if there is an easy way to do this and whether anyone had any
good ideas?
Current approach I am considering: Route the Log Files coming in – to a Split
Text & Route Text Processor to make sure no empty logs get through and get the
individual log count when files contain multiple logs – At the end of this the
total number of logs are visible in the UI queue, where it displays the
queueCount, but this information is not readily available to any processor.
Current thought process is that I can use the ExecuteScript Processor and
update a local file to keep track and insert the document into elastic search
hourly.
Any advice would be appreciated
Thanks,
Sai Peddy
________________________________________________________
The information contained in this e-mail is confidential and/or proprietary to
Capital One and/or its affiliates and may only be used solely in performance of
work or services for Capital One. The information transmitted herewith is
intended only for use by the individual or entity to which it is addressed. If
the reader of this message is not the intended recipient, you are hereby
notified that any review, retransmission, dissemination, distribution, copying
or other use of, or taking of any action in reliance upon this information is
strictly prohibited. If you have received this communication in error, please
contact the sender and delete the material from your computer.
