Hi Ali,

I observed ~1GB/sec on a test PutFile processor using an enterprise PCIe  NVMe 
ssd on a single instance on desktop class hardware.  I plan to run more in 
depth tests on server class hardware but will likely be on 1 Gb network. I 
should note I'm not sure exactly how much provenance was being written. 

The nifi-0.7.0  instance was a fresh install with no major configuration 
changes.  I was using the generate flowfiles processor to generate 100MB 
flowfiles and writing as fast as possible with a PutFile processor.

The SSD posted the following on the AS-SSD benchmark (completely unoptimized):
1.8GB/sec for sequential write;
2.3GB/sec for 4K random write (64 treads);
114MB/sec for 4K random write (1 thread)

On the PCie bus, you should easily surpass 800 MB/sec, especially if your 
flowfiles are large and you have few provenance events.  The theoretical 
bandwidth is 985 MB/sec/lane, up to 16 lanes; I was running x4.  The NVMe 
standard should also help with smaller flowfiles.  

Hope this helps,

> On Oct 17, 2016, at 12:06 PM, Ali Nazemian <alinazem...@gmail.com> wrote:
> Dear Nifi users/developers,
> Hi,
> I was wondering how can I calculate the theoretical throughput of a Nifi 
> server? let's suppose we can eliminate different bottlenecks such as the file 
> flow rep and provenance repo bottleneck by using a very high-end SSD.  
> Moreover, assume that a very high-end network infrastructure is available. In 
> this case, is it possible to reach 800MB throughput per second per each 
> server? Suppose each server comes with 24 disk slots. 16 disk slots are used 
> for creating 8 x RAID1(SAS 10k) mount points and are dedicated to the content 
> repo. Let's say each content repo can achieve 100 MB throughput. May I say 
> the total throughput per each server can be 8x100=800MBps?  Is it possible to 
> reach this amount of throughput practically?
> Thank you very much.
> Best regards,
> Ali

Reply via email to