Re: How do you use ElasticSearch with NiFi?

re already supported via > JsonQueryElasticSearch. If you didn't know that, please try it out and drop > some feedback on what is needed to make it work for your use cases.) > > Thanks, > > Mike > -- *Joe Percivall* linkedin.com/in/Percivall e: jperciv...@apache.com

Re: PutElasticsearchHttp can not use Flowfile attribute for ES_URL

the input JSON > document and saving the information inside a Flow attribute. > > > > What can I do about this? I don't like monkey patching.. is there any > other way to get around this? > > > > Thanks > -- *Joe Percivall* linkedin.com/in/Percivall e: jperciv...@apache.com

Re: Anyone using HashAttribute?

l/2980 > [2] https://github.com/apache/nifi/pull/2983 > > > > Andy LoPresto > alopre...@apache.org > *alopresto.apa...@gmail.com * > PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69 > > -- *Joe Percivall* linkedin.com/in/Percivall e: jperciv...@apache.com

Re: Best practices for running Apache NiFi in production in a Docker container

/questions/49854237/is-xxmaxramfraction-1-safe-for-production-in-a-containered-environment > > > On Thu, Aug 30, 2018 at 7:54 PM Joe Percivall > wrote: > >> Hey everyone, >> >> I was recently searching for a best practice guide for running a >> production instance o

Best practices for running Apache NiFi in production in a Docker container

uot;, has anyone used them in practice? [1] https://community.hortonworks.com/articles/7882/hdfnifi-best-practices-for-setting-up-a-high-perfo.html [2] https://developers.redhat.com/blog/2017/04/04/openjdk-and-containers/#more-433899 [3] https://blog.csanchez.org/2017/05/31/running-a-jvm-in-a-conta

Re: Prioritizing flowFiles to tailor throughput

al level - a lightweight NiFi server to > accelerate delivery of priority one content and email alerts, a priority > two suite of NiFi servers for standard flowfile volume, a priority three > suite of servers to handle long-term bulk processing, etc etc? > > Thanks in advance for your help. -Jim > -- *Joe Percivall* linkedin.com/in/Percivall e: jperciv...@apache.com

Re: Back Pressure Object threshold not honored

e to be set to a > small number of flowfiles to keep the source from flooding the destination. > Has anyone come across this problem before? I am running 12 instances of > NiFi on version 1.1.1 on Ubuntu 14.04. > > > > Regards, > > > > Kevin > > > -- *Joe Percivall* e: joeperciv...@gmail.com

Re: MiNiFi's differentiator

well for me. I see that the differentiator is configurable and > additional implementations can be provided. Are there any examples of > circumstances in which the using WholeConfigDifferentiator would not be the > best choice? > > Thanks, > Jeff > -- *Joe Percivall* linkedin.com/in/Percivall e: jperciv...@apache.com

Re: Expression language access to flowfile content?

far as i can >>>> tell doesn't have access to that flowfile content. >>>> >>>> Without having to put another processor in between monitor activity and >>>> put slack, is there a way to use the flowfile content directly from >>>> expression language? >>>> >>>> If not does anyone else think this would be really useful to make >>>> processors like these more compatible? >>>> >>> >>> >> > -- *Joe Percivall* linkedin.com/in/Percivall e: jperciv...@apache.com

Re: How to use Rest API ressource data-transfer?

>> Ismael >> >> >> >> -- >> View this message in context: http://apache-nifi-users-list. >> 2361937.n4.nabble.com/How-to-use-Rest-API-ressource-data-tra >> nsfer-tp666.html >> Sent from the Apache NiFi Users List mailing list archive at Nabble.com. >> > > -- *Joe Percivall* linkedin.com/in/Percivall e: jperciv...@apache.com

Re: Publish NiFi 1.1.1 to Maven

Hello Kevin, Thanks for emailing the users list. What problems exactly are you running into? The repository.apache.org is the Apache repo that holds the artifacts after they are released by the Release Manager. I released 1.1.1 in the same way as 0.7.1, 1.0.0 and 1.0.1 so I'm not sure where the

Re: Building nifi locally on my mac

Hello, I don't see any text between "... test error :" and "(on CentOS it works fine)". Could you try reformatting and resending? Joe On Thu, Dec 29, 2016 at 5:26 AM, ddewaele wrote: > When I try to create a local build of nifi on my mac I always get the > following test

New 'Powered by Apache NiFi' Website Section

Hello NiFi Users, Thanks to a contribution by Andre de Miranda we have a new "Powered By Apache NiFi" section on our webpage[1]. It would be great to expand this list out with many more companies and organizations that are using Apache NiFi. If you'd like to be added to this list please feel

Re: NiFi Cron scheduling

, 2016, 2:06:44 PM EST, Joe Percivall <joeperciv...@yahoo.com> wrote: I have created a JIRA here[1] for the issue. If you need a fix/workaround now (and are using the latest version), then you may want to utilize the newly added ability to use state with UpdateAttribute. You can set up

Re: NiFi Cron scheduling

, December 21, 2016, 9:04:32 AM EST, Joe Percivall <joeperciv...@yahoo.com> wrote: Totally didn't realized I accidentally moved this off of the mailing list. Brining it back. -- I just realized as well that NiFi is only using Quartz for the CronExpression. I think this bug

Re: NiFi Cron scheduling

ticket for this, or post this on the mailinglist ? On Tue, Dec 20, 2016 at 5:03 PM, Joe Percivall <joeperciv...@yahoo.com> wrote: > I am running on a Mac version 10.10.5. I pulled down the 1.1.0 zip, created a > flow with a GenerateFF -> ReplaceText -> LogAttribute (one for success an

Re: ReplaceText and special characters

This question actually gets back to a discussion on the user entering literal vs. escaped text. In the NiFi UI the user inputs the text into the box and then it is converted into a Java String which gets automatically escaped in order to pass along the string as the user wrote it (so a

Re: ExecuteStreamcommand

Hey Juan, I believe you need to escape the escape, so in order to do a double quote it would be \\" as the property value. Joe - - - - - - Joseph Percivall linkedin.com/in/Percivall e: joeperciv...@yahoo.com On Friday, December 16, 2016 2:05 PM, Juan Sequeiros

Re: Controller services visibility problem

Hello Panos, With the 1.0.0 update, security policies were added to process groups and controller level features. This means that there is now a difference between creating a Controller Service (CS) which is scoped to the controller (ie. for Reporting tasks) and creating a CS which is scoped to

Re: Getting the number of logs

Hello Sai, I'm gonna paraphrase what I think your use-case is first, let me know if this is wrong. You want to keep track of the number of logs coming in and every hour you want to document how many came in in that hour. Currently NiFi doesn't handle this type of "stateful" event processing

Re: MQTT Publisher errors

Nov 3, 2016 at 2:39 PM, Joe Percivall <joeperciv...@yahoo.com> wrote: Hello, > > >Sorry you are having problems with PublishMQTT. What MQTT broker are you >trying to hit? and could you pass along what configuration you have set >(including scheduling tab)? I'd like to tr

Re: MQTT Publisher errors

Hello, Sorry you are having problems with PublishMQTT. What MQTT broker are you trying to hit? and could you pass along what configuration you have set (including scheduling tab)? I'd like to try and reproduce if possible. I'm guessing there is nothing more to that stacktrace? I ask (hoping that

Re: Stable version of 1.x?

Hello John, Ah sorry, that mention of "Non-stable" is my fault. The current 1.0.0 release is stable. That note was for the 1.0.0-BETA release and I forgot to remove it when changing over to 1.0.0. It have been removed. Thank you for brining this up! Joe - - - - - - Joseph Percivall

Re: Processor that Decompresses Files?

Hello Keren, The "decompress" mode is an option of the CompressContent[1] processor and should solve your use-case. [1] https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi.processors.standard.CompressContent/index.html Joe - - - - - - Joseph Percivall

Re: Provenance expiration error

Hello Adam, Sorry no one has responded yet. Taking a look at the stack trace, I think you are running into NIFI-2087[1]. This was addressed in 1.0.0. [1] https://issues.apache.org/jira/browse/NIFI-2087 Joe - - - - - - Joseph Percivall linkedin.com/in/Percivall e: joeperciv...@yahoo.com

Re: Request for enhancement

- Moving users list to BCC Hello Gunjan, This seems like a good potential idea. The proper place to submit the suggestion is through the Apache NiFi Jira[1]. It can more easily be discussed and worked on there. [1] https://issues.apache.org/jira/browse/NIFI Suggestions/ideas from users are

Re: How to deal with decimals while they're not supported?

Hey Stephane, Currently working with Decimals in NiFi is like putting a square peg in a round hole. I haven't tried to do it much but for a simple use-case like yours I believe there are two options.  1: Use Expression Language and a Regex to move the decimal2: (Not 100% sure this works) Use

[ANNOUNCE] Apache NiFi 1.0.0-Beta release

Hello The Apache NiFi team would like to announce the release of Apache NiFi 1.0.0-BETA. The upcoming 1.0.0 release will mark the culmination of a lot of work over the last few months with many new framework level features being added. This Beta release was cut to give our Apache NiFi users a

Re: InvokeHTTP[xxxx] failed to process session due to java.lang.NoClassDefFoundError: android/util/Log:

Loader.loadClass(ClassLoader.java:424) ~[na:1.8.0_72] at java.lang.ClassLoader.loadClass(ClassLoader.java:357) ~[na:1.8.0_72] ... 11 common frames omitted On Tue, Jun 28, 2016 at 2:16 PM, Joe Percivall <joeperciv...@yahoo.com> wrote: Okhttp has elected to not have built-in support for Digest Auth[

Re: Replace Text

e: Yes that's exactly correct. > On Jun 13, 2016, at 11:14 AM, Joe Percivall <joeperciv...@yahoo.com> wrote: > > Sorry I got a bit confused, in your original question you said that you > wanted to append the value and I took it that you just wanted to append the > value to

Re: Replace Text

file and not to the one i waned too. On Mon, Jun 13, 2016 at 10:40 AM, Joe Percivall <joeperciv...@yahoo.com> wrote: Hello, > >In order to use ReplaceText[1] to solely append a value to the end of then >entire text then change the "Replacement Strategy" to "Append"

Re: Replace Text

Hello, In order to use ReplaceText[1] to solely append a value to the end of then entire text then change the "Replacement Strategy" to "Append" and leave "Evaluation Mode" as "Entire Text". This will take whatever is the "Replacement Value" and append it as a literal(without interpreting

Re: Processor Question

For number one, you can also use RouteText[1] with the matching strategy "Satisfies Expression". Then as a dynamic property use this expression "${lineNo:le(10)}". This will route first 10 lines to the "matched" relationship (assuming "Route to each matching Property Name" is not selected).

Re: Guidance for NiFi output streaming

Hello Stephane, Just to be sure I have your use-case correct, you are ingesting a continuous stream of lat/lon information for various devices. Every 1 second you want to take the information from the previous second and write out just the most recent lat/lon of each device. An important

Re: Doing development on nifi

Hello Stéphane, Just adding on to Matt's and Andy's answers, Andy mentioned Provenance[1] for replaying events but I also find it very useful for debugging processors/flows as well. Data Provenance is a core feature of NiFi and it allows you to see exactly what the FlowFile looked like

Re: Help with replace method

Hello Igor, I got your template working by using the below replacement string and changing the "Replacement Strategy" to "Always Replace". I've attached a template that works for me. {"test":"${teststr:replaceAll('"','"')}"} The backslashes are a bit weird because they escape characters

Re: prepend attribute value to flowfile content

Hello Sumo, Check out the "Replacement Strategy" property of the "ReplaceText" processor[1]. Setting this to "prepend", an "Evaluation Mode" of "Entire text" and a character set of UTF-8 you should be able to accomplish this.

Re: Help on creating that flow that requires processing attributes in a flow content but need to preserve the original flow content

Hello Chris, The EvaluateJsonPath processor has the property "Destination" which gives you the option to send it either to the FlowFile content or a FlowFile attribute. Selecting "flowfile-attribute" will place the value in the "kafka.key" attribute of the FlowFile. You can find documentation

Re: List Files

Hello, ListFile is a source processor so this behavior is expected. It supports expression language so that it can be configured to utilize certains methods. For example, some people may want to get a list of files from a rotating directory that gets created every hour. To do that they would

Re: How to configure a ExecuteStreamCommand

atch file (there is no new file called foo.txt on my desktop). > > Have you any idea what I am doing wrong? I am pretty sure should be an easy > fix. Please fell free to make any comment or suggestion regarding to my case. > > Thanks in advance. > >> El 2 mar 2016, a la

Re: How to configure a ExecuteStreamCommand

Hello, Welcome to NiFi! I just tried running an ExecuteStreamCommand processor with the properties you have (I created a script and modified the paths to point to a folder that exists) and two things jump out. One, the Max attribute length must take an integer. If you set it to be a path the

Re: Processor with State

On Wednesday, March 2, 2016 11:19 AM, Joe Percivall <joeperciv...@yahoo.com> wrote: Hello Claudio, Your use-case actually could leverage a couple of recently added features to create a really cool open-source processor. The two key features that were added are State Management and the a

Re: Processor with State

Hello Claudio, Your use-case actually could leverage a couple of recently added features to create a really cool open-source processor. The two key features that were added are State Management and the ability to reference processor specific variables in expression language. You can take a

Re: Maximum attribute size

Hello Lars, You are correct that the WAL is different from swapping.  Swapping is used when a single connection queue grows to be very large. A chunk of the FlowFiles are then swapped out of JVM memory and written to disk. Where they are stored until they are swapped back in for processing. The

Re: Log4j/logback parser via syslog

Hello Madhu, If you're looking for a template to show how to create a dynamic property for RouteOnAttribute to use, I'd suggest checking out this template[1]. It is a simple template that checks to see if the an attribute matches 'NiFi'. Also provenance can be a very powerful debugging tool.

Re: File Upload to a ListenHTTP Processor

Hello Andrew, I believe I was running into something similar before and my problem was actually with the curl command itself. When I added "--data-binary" to the command it worked. Relevant SO question: http://stackoverflow.com/questions/9134003/binary-data-posting-with-curl Joe - - - - - -

Re: Add date in the flow file attribute

Hello Sudeep, How precise do you need the date/time to be? What you could do is add an UpdateAttribute processor[1] after ingesting which uses the Expression language functions "now" [2] and "format" [3] to add the date/time down to the millisecond. There would of course be a bit of error

Re: Add date in the flow file attribute

ld see an AttributeToJson object but it seems to be appending the information in flow file content or attribute. What is a good way to capture only attributes and send it to MongoDb? On Tue, Feb 2, 2016 at 8:42 PM, Joe Percivall <joeperciv...@yahoo.com> wrote: Hello Sudeep, > >How precise do you

Re: JDBC External Table File Target

Hello Obaid, Sorry no one has gotten back to you sooner, many of the developers are working diligently to get 0.5.0 done. I don't know too much about loading SQL tables but when you say "local file system filename", is this a file that exits on the target system? If so you may just be able to

Re: Trying to send a PostHTTP with a "non-chunked" payload.

Hello Richard, This is actually came up a couple days ago and is a known bug [1]. In another ticket[2] I added the option to InvokeHttp expose the option and my patch is pending. [1] https://issues.apache.org/jira/browse/NIFI-1396[2]  https://issues.apache.org/jira/browse/NIFI-1405 Joe- - - - -

Re: how to sort json array in dataflow

Hello Roland, Were you able to achieve any success incorporating the JSON sorting into a custom processor? Joe - - - - - - Joseph Percivall linkedin.com/in/Percivall e: joeperciv...@yahoo.com On Monday, January 11, 2016 2:03 AM, 彭光裕 wrote: Hello Joe, Thanks

Re: Is there a way to configure a processor to run only N times

Hello Lars and Sudeep, I created a Jira ticket for this issue:  https://issues.apache.org/jira/browse/NIFI-1407 We can continue the conversation there instead two different email threads. Joe - - - - - - Joseph Percivalllinkedin.com/in/Percivalle: joeperciv...@yahoo.com On Thursday,

Re: PutDistributedMapCache

to build the code for only a particular processor? Just curious if we can build and deploy a particular processor in an existing NiFi environment. On Wed, Jan 13, 2016 at 9:33 PM, sudeep mishra <sudeepshekh...@gmail.com> wrote: Thanks Joe. I will try out the patch. On Wed, Jan 13, 2016 a

Re: PutSQL question

Hello Ralph, A common way to replace the contents of a FlowFile with specific text is to use the ReplaceText processor. The default search query will match the entire content and replace it with whatever your replacement value is. So if you set the replacement value (complete with expression

Re: PutDistributedMapCache

ecific configuration as of now as I am still exploring NiFi. Though I think it would be helpful to let user store and retrieve the cache values in different formats json, avro etc. Thanks & Regards, Sudeep On Tue, Jan 12, 2016 at 9:15 PM, Joe Percivall <joeperciv...@yahoo.com> wrot

Re: PutDistributedMapCache

Hello Sudeep, We are currently lacking a "GetDistributedMapCache" processor that corresponds to the "PutDistributedMapCache". I created a ticket[1] and will be working on it today. If you have any comments, configuration suggestions, etc. please let me know or comment on the ticket. [1] 

Re: Is it possible to to create a dynamic Processor through API call from PHP

>>request. The thing is i receive them with random names, i'll try to find out >>how i can parse them to rename them as JSON files instead of POST fields. >>Thanks for your help. >> >> >>On 29 December 2015 at 16:50, Joe Percivall <joeperciv...@yahoo.com

Re: Is it possible to to create a dynamic Processor through API call from PHP

Hello Kacem, There are multiple different ways to get information into NiFi. ListenHTTP would be the easiest way to start an HTTP Server that is used to receive FlowFiles from remote sources. Assuming your PHP code only acts as a router, ListenHTTP should allow you to bypass the PHP code

Re: InvokeHTTP request parameters

Hello James, I'm actually working on repo that includes many different templates. There are a few that have InvokeHttp in them (can find the exact ones using the excel doc at the top level) but this is a simple one that uses InvokeHttp:

Fw: [RMX:NL] Re: [RMX:NL] Re: [RMX:NL] Re: InvokeHTTP request parameters

I accidentally replied to just James with one of my responses. Forwarding the thread back. - - - - - - Joseph Percivall linkedin.com/in/Percivall e: joeperciv...@yahoo.com On Friday, December 18, 2015 5:20 PM, Joe Percivall <joeperciv...@yahoo.com> wrote: Following this post[1] on s

Re: [RMX:NL] Re: [RMX:NL] Re: InvokeHTTP request parameters

used that template, but changed the remote URL to point to my server (https://www.google.com -> http://localhost). James -----Original Message- From: Joe Percivall [mailto:joeperciv...@yahoo.com] Sent: Friday, December 18, 2015 1:25 PM To: users@nifi.apache.org Subject: [RMX:NL] Re: [RMX:NL] Re: I

Re: Content replacement

Hey Chandu, You messed up bit in the config of your ReplaceText processor. You should be running "Line-by-line" mode so each line is treated separately.  Also your search value is a bit messed up. It was trying to use capture groups in it ("$2"), it wasn't successfully handling an initial date

Re: Mirroring nifi lists on gmane

Does anyone have any more opinions on this? - - - - - - Joseph Percivalllinkedin.com/in/Percivalle: joeperciv...@yahoo.com On Saturday, December 5, 2015 6:54 AM, Nigel Jones wrote: I have no experience of nabble but I'm also open to any similar suggestions? . 

Re: InvokeHTTP Content Type

ot; along with the request using CURL for example. I tried to set that on the "Attributes to Send" property in the InvokeHTTP processor, but it did not work. It always returns XML. Thanks, Indus On Fri, Dec 4, 2015 at 2:49 PM, Joe Percivall <joeperciv...@yahoo.com> wrote: Hello Indus, >

Re: InvokeHTTP Content Type

Hello Indus, Are you receiving a specific error when you attempt to receive a JSON response? I can configure an InvokeHTTP processor to GET "http://api.randomuser.me/; by just setting the URL and it properly receives back a JSON response. Joe - - - - - - Joseph Percivall

Re: InvokeHTTP Content Type

on' is invalid because 'Accept' is not a supported property." Did I do something wrong? Thanks, Indus On Fri, Dec 4, 2015 at 3:12 PM, Joe Percivall <joeperciv...@yahoo.com> wrote: I see, as for the request I'm guessing you mean you pass a header with key "Accept" and val

Re: Is it possible to create instances of a processor dynamically based on HTTP response?

is the comments example. i have many people who commented the post, i need to do an HTTP request for each one. So i need to invoke not one HTTP request but many after each retrieval of one post comment. Kindly. ​ On 1 December 2015 at 17:07, Joe Percivall <joeperciv...@yahoo.com> wrote: He

Re: Is it possible to create instances of a processor dynamically based on HTTP response?

Hello Kacem, For your first question you can set up a relatively simple flow to achieve this. Assuming the first retrieval is a GET method call you can do: GetHTTP -> ExtractText -> InvokeHTTP. What this does is, it first gets the JSON files from the social network using GetHTTP. Then with

Re: another proxied question from irc

Hey Tony, I can't speak to the best practice for this controller service but just wanted to follow up to see if this user got their question answered. Joe - - - - - - Joseph Percivall linkedin.com/in/Percivall e: joeperciv...@yahoo.com On Monday, November 23, 2015 3:32 PM, Tony Kurc

Re: Encrypting Attributes

tant now. I want to see how Nifi can be useful here like encrpyting only few fields before processing. Is it possible to create a new flow file with those fields, route and encrypt that flow file? Thanks, Naveen On 11/30/15, 11:28 AM, "Joe Percivall" <joeperciv...@yahoo.com>

Re: queued files

Hello Charlie, I was looking back through and saw this wasn't totally resolved yet. Couple questions. First, what system are you using? There are a couple of options for the stream command depending on what you're using. Also are you able to get new commands (using yum or brew)? The key

Re: High CPU usage in FileSystemRepository.java

Hello Adam, Are you still seeing high cpu usage? Sorry no has gotten back to you sooner, we are all working very hard to get 0.4.0 out. Joe - - - - - - Joseph Percivall linkedin.com/in/Percivall e: joeperciv...@yahoo.com On Friday, November 13, 2015 4:10 PM, Adam Lamar

Re: Example for mongo processors

Hello Subhash, Sorry no one has gotten back to you sooner but what kind of examples are you looking for? Are you looking for examples of how they are configured, a template or just cool applications? Joe - - - - - - Joseph Percivall linkedin.com/in/Percivall e: joeperciv...@yahoo.com

Re: Nifi - InvokeHTTP - retrying on 3xx with Location header issue

:56 PM, M Singh <mans2si...@yahoo.com> wrote: Joe: No problem and thanks for your support in fixing the issue. Mans On Monday, November 9, 2015 12:35 PM, Joe Percivall <joeperciv...@yahoo.com> wrote: Mans, I did a bit of testing using my refactor and I understand what the

Re: Nifi - InvokeHTTP - retrying on 3xx with Location header issue

is up, Joe - - - - - - Joseph Percivall linkedin.com/in/Percivall e: joeperciv...@yahoo.com On Monday, November 9, 2015 2:05 PM, Joe Percivall <joeperciv...@yahoo.com> wrote: Hey Mans, Thanks for creating the ticket. First, have you tried the patch I supplied in 1086 [1]? I changed the unde

Re: Help me figure out why this wont work

Hey Chris, I feel your "Ugg" and I'd be happy to knock it out. I'm doing a bunch of work with InvokeHttp already so I should be able to get this done pretty quick. Joe- - - - - - Joseph Percivall linkedin.com/in/Percivall e: joeperciv...@yahoo.com On Monday, November 9, 2015 3:23 PM,

Re: Help me figure out why this wont work

linkedin.com/in/Percivall e: joeperciv...@yahoo.com On Monday, November 9, 2015 3:26 PM, Joe Percivall <joeperciv...@yahoo.com> wrote: Hey Chris, I feel your "Ugg" and I'd be happy to knock it out. I'm doing a bunch of work with InvokeHttp already so I should be able to get

Re: ConvertCharacterSet

Hey Charlie, Sorry no one has followed up with you yet. One way I see around ConvertCharacterSet not supporting expression language is to route on attribute (assuming the character set is extracted to be an attribute) to different ConvertCharacterSet processors depending on the input character

Re: Nifi Clustering - work distribution on workers

Hey Mans, To load balance and send the FlowFiles to a processing group or processor in the same cluster you will still set up an RPG but you will just point the RPG to the current cluster's NCM. The NiFi Cluster Manager (NCM) is in charge of splitting the incoming FlowFiles among the nodes.

Re: Feeding & Consuming data to & from Nifi

Hey Krish, Welcome to NiFi! Storm is how I got my start with data flow as well and NiFi has been awesome. NiFi's "processors" are equivalent to Storm's "spouts" and "bolts". NiFi can certainly handle the logging/auditing use case among others. With NiFi's data provenance you can see exactly how