Hello Jon, The number of processors is virtually unlimited, provided that you have enough CPU to sustain the number of concurrent tasks allocated to NiFi to provide acceptable performance, enough disk space for the NiFi repositories, and enough RAM to cover the overhead of the processor instances. With 1,000 processors, you'll need enough concurrent tasks to keep the latency of data flow to an acceptably low level. Depending on the processors you're using, and the design of your flow, you may need to configure NiFi with additional heap space to cover the requirements of that many processors.
NiFi 1.x had added quite a few features that are not available in the 0.x version, like multi-tenancy support, additional user authentication options (OpenID/Knox/improved LDAP configuration), improved proxy support, NiFi Registry support, new processors, increased security, and bug fixes, among other things. Take a look at some of the release notes from the 1.x line [1] for more details, There are improved repository implementations, along with Record-based processors that improve throughput of data through NiFi, in the 1.x line that I don't think were back-ported to 0.x. Are you running a single instance (non-clustered) of NiFi? You may want to create a NiFi cluster to set up distributed processing of data if the resources of the host on which you're running NiFi is not able to keep up. How has your current configuration been performing? I know this answer is very high-level, and would be happy to dive into some details if you'd like. [1] https://cwiki.apache.org/confluence/display/NIFI/Release+Notes On Mon, Jan 29, 2018 at 5:24 PM Jon Allen <[email protected]> wrote: > Is there a limit to the number of processors that can be put into a NiFi > flow? > > I'm assuming there isn't an actual hard limit but what's likely to break > as the number of processors increases, and what's considered a large number? > > We currently have a few hundred processors in our graph but it's looking > like this will head towards 1,000 in the near future. Does anyone have any > suggestions for tuning the system to handle this? Are there any papers > available describing what I should be looking at? > > We're currently running with NiFi 0.7.4. Are there any changes in later > releases that improve things in this area? > > Thanks, > Jon >
