Christian, Another factor to consider for the NiFi nodes is the disk hardware and configuration. Simply put, the faster your disks are (SSD, RAID0, etc), the faster NiFi will be able to perform. This is dependent on the needs of your flow; the number of CPUs/cores isn't the only consideration for performance. NiFi can use multiple repositories (for content, flowfiles, etc) to allow "striping" across multiple locations (directories, mount points, separate physical disks, etc) which can help alleviate disk IO utilization concerns.
It may help you to create a prototype flow on currently available hardware to take a look at CPU/IO/disk utilization to get a starting point on what hardware you'll need. On Sat, Jun 1, 2019 at 3:12 AM Christian Andreasen < [email protected]> wrote: > Thank you for your input, Martijn. It def helps us in our decision-making. > > Den tor. 30. maj 2019 kl. 11.21 skrev Martijn Dekkers < > [email protected]>: > >> In clusters, odd numbers of nodes are generally preferred (depending on >> clustering implementation) to avoid (to an extent) split-brain scenarios >> and generally manage quorum. I stand to be corrected, but in the current >> implementation in NiFi I don't think this is an issue. >> >> Additional nodes will give you increased IO throughput for most cases. IO >> will, in most cases, be your bottleneck. >> Core/thread count per node will have an impact on scheduling. Matt Clarke >> wrote an excellent article on thread usage in NiFi: >> https://community.hortonworks.com/articles/221808/understanding-nifi-max-thread-pools-and-processor.html >> >> "Optimal" Cluster design will come down to your anticipated use-cases. >> Having said that, most run-of-the-mill "decent" systems will deliver great >> performance for most systems. If your needs are more towards the "we *must* >> have very high performance", or "we *must* process x messages per second" >> to a degree of business criticality, you should probably make sure you >> design your flow and then design and implement a system to meet the needs >> of that flow. >> >> On Thu, 30 May 2019, at 09:52, Christian Andreasen wrote: >> >> We are planning to build a NiFi cluster and have two questions that we >> hope you could help us answer. >> >> 1. When having our NiFi cluster configured to run with an external >> Zookeeper cluster (i.e. not using the default embedded ZK mode) is it then >> still best practice to have an odd number of NiFi nodes? If so, why is >> that? >> 2. Keeping all other things constant, is there then any advantage of >> running a setup with 3 NiFi nodes each having 8 cores compared to a setup >> with 6 nodes each having 4 cores? >> >> Any input much appreciated. >> >> Thanks, >> Christian >> >> >>
