Sai, I’d suggest you look at using a ControllerStatusReportingTask [1], which monitors the processor and provides statistics from that component. If you need to use this data within NiFi, you can also use SiteToSiteProvenanceReportingTask [2], which can export provenance events as data that can be consumed by (the same or a different) instance of NiFi. Both of these may be overkill for your use case (the provenance reporting task will offload all of the provenance events from the application), and if so, you may be able to use counters [3] to do this quickly and easily (but be aware that the values are just held in memory, so if you’re writing to ES hourly, you should be ok, but they won’t persist across restart). Your initial thought to use ExecuteScript would also work.
I believe Joe Percivall had done some work on SEP and window/aggregate calculations before. That may also help with what you are doing. Joe Percivall·4:51 PM Here's the link to the processor: https://github.com/JPercivall/nifi/blob/newRollingState/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/RollingWindowAggregator.java Here's the ticket: https://issues.apache.org/jira/browse/NIFI-1682?jql=project%20%3D%20NIFI%20AND%20text%20~%20%22rolling%20window%22 Keep in mind that this work is old and will need to be updated. I do have a pending PR for UpdateAttribute with State though: https://github.com/apache/nifi/pull/319 [1] https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi.controller.ControllerStatusReportingTask/index.html <https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi.controller.ControllerStatusReportingTask/index.html> [2] https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi.reporting.SiteToSiteProvenanceReportingTask/index.html <https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi.reporting.SiteToSiteProvenanceReportingTask/index.html> [3] https://community.hortonworks.com/questions/50622/apache-nifi-what-are-counters-in-nifi.html <https://community.hortonworks.com/questions/50622/apache-nifi-what-are-counters-in-nifi.html> Andy LoPresto [email protected] [email protected] PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69 > On Nov 9, 2016, at 8:14 AM, Peddy, Sai <[email protected]> wrote: > > Hi All, > > I’m currently working on a use case to be able to track the number of > individual logs that come in and put that information in ElasticSearch. I > wanted to see if there is an easy way to do this and whether anyone had any > good ideas? > > Current approach I am considering: Route the Log Files coming in – to a Split > Text & Route Text Processor to make sure no empty logs get through and get > the individual log count when files contain multiple logs – At the end of > this the total number of logs are visible in the UI queue, where it displays > the queueCount, but this information is not readily available to any > processor. Current thought process is that I can use the ExecuteScript > Processor and update a local file to keep track and insert the document into > elastic search hourly. > > Any advice would be appreciated > > Thanks, > Sai Peddy > > ________________________________________________________ > > The information contained in this e-mail is confidential and/or proprietary > to Capital One and/or its affiliates and may only be used solely in > performance of work or services for Capital One. The information transmitted > herewith is intended only for use by the individual or entity to which it is > addressed. If the reader of this message is not the intended recipient, you > are hereby notified that any review, retransmission, dissemination, > distribution, copying or other use of, or taking of any action in reliance > upon this information is strictly prohibited. If you have received this > communication in error, please contact the sender and delete the material > from your computer.
signature.asc
Description: Message signed with OpenPGP using GPGMail
