Hi Ali, I observed ~1GB/sec on a test PutFile processor using an enterprise PCIe NVMe ssd on a single instance on desktop class hardware. I plan to run more in depth tests on server class hardware but will likely be on 1 Gb network. I should note I'm not sure exactly how much provenance was being written.
The nifi-0.7.0 instance was a fresh install with no major configuration changes. I was using the generate flowfiles processor to generate 100MB flowfiles and writing as fast as possible with a PutFile processor. The SSD posted the following on the AS-SSD benchmark (completely unoptimized): 1.8GB/sec for sequential write; 2.3GB/sec for 4K random write (64 treads); 114MB/sec for 4K random write (1 thread) On the PCie bus, you should easily surpass 800 MB/sec, especially if your flowfiles are large and you have few provenance events. The theoretical bandwidth is 985 MB/sec/lane, up to 16 lanes; I was running x4. The NVMe standard should also help with smaller flowfiles. Hope this helps, Lee > On Oct 17, 2016, at 12:06 PM, Ali Nazemian <alinazem...@gmail.com> wrote: > > Dear Nifi users/developers, > Hi, > > I was wondering how can I calculate the theoretical throughput of a Nifi > server? let's suppose we can eliminate different bottlenecks such as the file > flow rep and provenance repo bottleneck by using a very high-end SSD. > Moreover, assume that a very high-end network infrastructure is available. In > this case, is it possible to reach 800MB throughput per second per each > server? Suppose each server comes with 24 disk slots. 16 disk slots are used > for creating 8 x RAID1(SAS 10k) mount points and are dedicated to the content > repo. Let's say each content repo can achieve 100 MB throughput. May I say > the total throughput per each server can be 8x100=800MBps? Is it possible to > reach this amount of throughput practically? > Thank you very much. > > Best regards, > Ali
