Your scenario brings up an interesting point... back-pressure can only help you with in NiFi, meaning that in a push scenario, NiFi has no way of telling the syslog server (or any external source) to stop sending messages.
The way the syslog processor works, there is a background thread receiving the messages and placing them on an internal queue, and the processor when it runs is pulling messages off the internal queue [1]. So if you applied back-pressure and the processor stopped running, the processor would stop pulling messages off the queue, but the messages would still be coming and being placed in the queue until it filled up, and at some point data would start getting dropped. You may be able to tune the ListenSyslog processor such that you can avoid needing back-pressure, based on some of the recommendations in the post below. [1] https://community.hortonworks.com/articles/30424/optimizing-performance-of-apache-nifis-network-lis.html On Tue, May 31, 2016 at 11:06 AM, pradeepbill <[email protected]> wrote: > Thanks Bryan, helps a lot, > here is my use case, I use a listen Syslog processor -> Data Output > port(Spark application), now from the back pressure discussion, assuming I > am using “Back pressure data size > threshold.”=100MB , and that size is reached, and listen Syslog processor > wont run and , but still the real source from where the listen Syslog > processor is listening is emitting data , what happens to that data ?. lost > ?. > > > > -- > View this message in context: > http://apache-nifi-developer-list.39713.n7.nabble.com/back-pressure-tp10801p10803.html > Sent from the Apache NiFi Developer List mailing list archive at > Nabble.com. >
