Eric There is a practical limit in terms of memory, browser performance, etc... But there isn't otherwise any real hard limit set. We've seen flows with many 10s of thousands of processors that are part of what can be many dozens or hundreds of process groups. But the challenge that comes up is less about the number of components and the sheer reality of running that many different flows within a single host system. Now sometimes people doing flows like that don't have actual live/high volume streams through all of those all the time. Often that is used for more job/scheduled type flows that run periodically. That is different and can work out depending on time slicing/etc..
The entire notion of how NiFi's clustering is designed and works is based on 'every node in the clustering being capable of doing any of the designed flows'. We do not have a design whereby we'd deploy certain flows on certain nodes such that other nodes wouldn't even know they exist. However, of course partitioning the work to do be done across a cluster is a very common thing. For that we have concepts like 'primary node only' execution. Concepts like load balanced connections with attribute based affinity so that all data with a matching attribute end up on the same node/etc.. It would be very interesting to understand more about your use case whereby you end up with 100s of thousands of processors and would want node slicing of flows in the cluster. Thanks On Wed, Mar 8, 2023 at 9:31 AM Eric Secules <[email protected]> wrote: > > Hello, > > Is there any upper limit on the number of processors that I can have in my > nifi canvas? Would 100000 still be okay? As I understand it, each processor > takes up space on the heap as an instance of a class. > > If this is a problem my idea would be to use multiple unclustered nifi nodes > and spread the flows evenly over them. > > It would be nice if I could use nifi clustering and set a maximum replication > factor on a process group so that the flow inside it only executes on one or > two of my clustered nifi nodes. > > Thanks, > Eric
