Thanks for the pointer, Joe! I will be going through them. My exact problem is I am trying to think if I can trace a message through the system; something like a message entering NiFi, leaving it, and then reentering it, & NiFi recognising that this is the same flow that is still going through. Consider the scenario:
1. HTTP request coming in through a web server. 2. This goes to backend for processing & a copy (or some attribute to ID this request) goes to NiFi. 3. The backend processes this request & then sends a copy (or the attribute identifying the message & its response) to NiFi. 4. Web server responds to the request. 5. In NiFi, I should be able to trace the request from web-server to backend to response. Is this possible? Can NiFi link the different messages it receives at different times based on some unique ID? -- κρισhναν On Sun, Sep 20, 2015 at 9:51 PM, Joe Percivall <[email protected]> wrote: > Hey Krish, > > Welcome to NiFi! Storm is how I got my start with data flow as well and > NiFi has been awesome. NiFi's "processors" are equivalent to Storm's > "spouts" and "bolts". > > NiFi can certainly handle the logging/auditing use case among others. With > NiFi's data provenance you can see exactly how each message was handled, > the content and attributes at each stage, as well as replay any message at > any stage if you feel the need. > > For ingesting into NiFi there are a couple of options depending on your > system. If the files already exist on the same system then there is a > simple "getFile" processor. For other set-ups you can check out the "get" > processors in the documentation [1]. For forwarding to another service > again it depends on your system. If you set up two NiFi instances you can > check out "Remote Process Group". If you're just "Put"ing it through HTTP > there is a processor for that as well. Again there are other ways for > exfiltrate the data depending on your set-up. > > So for the configuration it all depends on what you want to do and how > your system is set up. If you give a bit more information on the specific > services you're using or use-case we can give even more direction. > > [1] https://nifi.apache.org/docs.html > > Again welcome to NiFi, > Joe > - - - - - - > *Joseph Percivall* > linkedin.com/in/Percivall > e: [email protected] > > > > > On Sunday, September 20, 2015 1:15 AM, Krish <[email protected]> > wrote: > > > Hi, > > I am a n00b at NiFi although I have worked with Storm; currently that is > how we handle data flow logic. > > I am evaluating using NiFi for logging/auditing use case (might move to > other uses if this works) in a ~50 machine cluster, and am thinking if it > would be a good fit to ingest messages from various sources and spit out > the graph of how a message was handled at each stage, by each of the > services. > > Anyway, I was wondering how do we feed data into a NiFi cluster to get the > first stage started. Also, how does the data exit the system, if I want it > to be forwarded to another service after NiFi is done with its processing > [similar to spouts and bolts in apache storm]. > > Thanks. > -- > κρισhναν > > >
