Hey Krish, Welcome to NiFi! Storm is how I got my start with data flow as well and NiFi has been awesome. NiFi's "processors" are equivalent to Storm's "spouts" and "bolts". NiFi can certainly handle the logging/auditing use case among others. With NiFi's data provenance you can see exactly how each message was handled, the content and attributes at each stage, as well as replay any message at any stage if you feel the need. For ingesting into NiFi there are a couple of options depending on your system. If the files already exist on the same system then there is a simple "getFile" processor. For other set-ups you can check out the "get" processors in the documentation [1]. For forwarding to another service again it depends on your system. If you set up two NiFi instances you can check out "Remote Process Group". If you're just "Put"ing it through HTTP there is a processor for that as well. Again there are other ways for exfiltrate the data depending on your set-up. So for the configuration it all depends on what you want to do and how your system is set up. If you give a bit more information on the specific services you're using or use-case we can give even more direction. [1] https://nifi.apache.org/docs.html Again welcome to NiFi,Joe- - - - - - Joseph Percivalllinkedin.com/in/Percivalle: joeperciv...@yahoo.com
On Sunday, September 20, 2015 1:15 AM, Krish <krishnan.k.i...@gmail.com> wrote: Hi, I am a n00b at NiFi although I have worked with Storm; currently that is how we handle data flow logic. I am evaluating using NiFi for logging/auditing use case (might move to other uses if this works) in a ~50 machine cluster, and am thinking if it would be a good fit to ingest messages from various sources and spit out the graph of how a message was handled at each stage, by each of the services. Anyway, I was wondering how do we feed data into a NiFi cluster to get the first stage started. Also, how does the data exit the system, if I want it to be forwarded to another service after NiFi is done with its processing [similar to spouts and bolts in apache storm]. Thanks.-- κρισhναν