I agree with everything Joe said, but just wanted to elaborate on one point...
Currently the way ListenSyslog is implemented [1], UDP and TCP both share the same logic that offers a message to an internal blocking queue with a 100 ms wait [2]. We should consider changing this for TCP so that it blocks indefinitely (or maybe just way longer) so that the client (the syslog server) wouldn't be able to send more data. With the current way it works you would not want to set back-pressure between ListenSyslog and whatever comes next, because this will cause ListenSyslog to stop processing messages on the internal queue, but the background thread will keep queueing them and eventually start dropping messages (the code in EventQueue below). [1] https://community.hortonworks.com/articles/30424/optimizing-performance-of-apache-nifis-network-lis.html [2] https://github.com/apache/nifi/blob/master/nifi-commons/nifi-processor-utilities/src/main/java/org/apache/nifi/processor/util/listen/event/EventQueue.java#L60 On Wed, Jul 6, 2016 at 4:32 PM, Joe Witt <[email protected]> wrote: > Hello > > Is this syslog via UDP or is it via TCP? If it is UDP then there > isn't much we can do to push back on the sender. If it is TCP then we > certainly will stop bringing in new data which in turn should > effectively push back but I'm not sure that will work well because the > sender may not be able to buffer long. Another consideration is that > NiFi itself can be used as a rather large buffer for this data itself. > > Alternatively, you might find that this is a good use case for Kafka > as a buffer where it is syslogsender -> nifi -> kafka and then the > spark jobs source data from kafka. That won't offer back pressure but > achieving true back pressure in syslog based sources broadly might be > elusive anyway. > > Thanks > Joe > > > On Wed, Jul 6, 2016 at 12:14 PM, pradeepbill <[email protected]> > wrote: > > Thanks David, I have done that, and I still see a connection being > > established between the ListenSysLog port and the server that is pushing > > data to ListenSysLog, then I thought NIFI is ignoring/loosing data after > > enabling back pressure, am I wrong?, I would like to be wrong ? Please > > advice. > > > > > > > > -- > > View this message in context: > http://apache-nifi-developer-list.39713.n7.nabble.com/conditional-Listen-Syslog-tp12652p12655.html > > Sent from the Apache NiFi Developer List mailing list archive at > Nabble.com. >
