Also, just wanted to clarify... I didn't mean to make it sound like back-pressure can't help with external sources, it really depends on the processor and protocol being used.
I was thinking mostly of ListenSyslog with UDP, and since UDP is connectionless back-pressure can't really help here. There are many other cases where it could help though. On Tue, May 31, 2016 at 1:01 PM, <[email protected]> wrote: > great!, testing my flow from your article recommendations, thanks again > Bryan. > > <quote author='Bryan Bende'> > Your scenario brings up an interesting point... back-pressure can only help > you with in NiFi, meaning that in a push scenario, NiFi has no way of > telling the syslog server (or any external source) to stop sending > messages. > > The way the syslog processor works, there is a background thread receiving > the messages and placing them on an internal queue, and the processor when > it runs is pulling messages off the internal queue [1]. > So if you applied back-pressure and the processor stopped running, the > processor would stop pulling messages off the queue, but the messages would > still be coming and being placed in the queue until it filled up, and at > some point data would start getting dropped. > > You may be able to tune the ListenSyslog processor such that you can avoid > needing back-pressure, based on some of the recommendations in the post > below. > > [1] > > https://community.hortonworks.com/articles/30424/optimizing-performance-of-apache-nifis-network-lis.html > > > On Tue, May 31, 2016 at 11:06 AM, pradeepbill <[email protected]> > wrote: > > > Thanks Bryan, helps a lot, > > here is my use case, I use a listen Syslog processor -> Data Output > > port(Spark application), now from the back pressure discussion, assuming > I > > am using “Back pressure data size > > threshold.”=100MB , and that size is reached, and listen Syslog > processor > > wont run and , but still the real source from where the listen Syslog > > processor is listening is emitting data , what happens to that data ?. > > lost > > ?. > > > > > > > > -- > > View this message in context: > > > http://apache-nifi-developer-list.39713.n7.nabble.com/back-pressure-tp10801p10803.html > > Sent from the Apache NiFi Developer List mailing list archive at > > Nabble.com. > > > > </quote> > Quoted from: > > http://apache-nifi-developer-list.39713.n7.nabble.com/back-pressure-tp10801p10804.html > > > _____________________________________ > Sent from http://apache-nifi-developer-list.39713.n7.nabble.com > >
