HI Clay, So as Bryan has said the actual connection is managed by a selector and all this does is goes through each connection and once that connection has data to receive it the selector then hands that over to a thread in the TCP receiving thread pool which does then some basic TCP processing and puts it into a buffer for an instance of associated ListenSyslog processor to processes, when the framework executes an instance of that processor.
Just so you're aware while setting the maximum number of connections does create a thread pool of 4,000 threads. In reality these threads don't really exist until one is created by the selector to run on the pool. So in short unless a single Nifi server gets 4,000 syslog messages in a very short space time (< 1 micro-second) I can't see it being an issue. Edward On Fri, Aug 2, 2019 at 2:06 PM Bryan Bende <[email protected]> wrote: > The actual connections themselves are managed with a selector, so if > all the connections are idle there should only be one thread for the > socket. > > As soon as a connection has something available to read then a thread > is spawned to start reading the connection until either no matter is > available, or it is closed. > > On Fri, Aug 2, 2019 at 7:18 AM Clay Teahouse <[email protected]> > wrote: > > > > Hello Edward, > > So, if have of to listen to 32,000 tcp connections and I have only 80 > cores, and I configure each ListenSyslog instance for 4,000 connections, > doesn't each spawn 4,000 threads behind the scene? The tcp connections will > be idle most of the time. > > > > thanks > > Clay > > > > > > On Fri, Aug 2, 2019 at 6:10 AM Edward Armes <[email protected]> > wrote: > >> > >> Hi Clay, > >> > >> Because Nifi underneath uses a thread pool for it's own threading > underneath, and each instance processor runs does so in it's own thread, I > don't see any reason why not. One thing to note that the way the ListenTCP > processor appears to have been written such that it gets all the requests > that have been received on that socket and processes them until either it > has no more requests left or process or that instance of the processor is > no longer scheduled to run. > >> > >> Hope that helps > >> > >> Edward > >> > >> On Fri, Aug 2, 2019 at 11:28 AM Clay Teahouse <[email protected]> > wrote: > >>> > >>> Hello All, > >>> > >>> I need to listen to and process thousands of persistent TCP > connections. I have 10 nodes, each having 8 cores. > >>> My understanding is that with existing NiFi listening processors, such > as ListnSyslog, a thread is utilized for each TCP connection. Does this > scale? Do I need to write a custom processor that utilizes a thread pool > for reading the data from the socket and processing them? > >>> > >>> thanks > >>> Clay >
