I observed ~1GB/sec on a test PutFile processor using an enterprise PCIe NVMe
ssd on a single instance on desktop class hardware. I plan to run more in
depth tests on server class hardware but will likely be on 1 Gb network. I
should note I'm not sure exactly how much provenance was being written.
The nifi-0.7.0 instance was a fresh install with no major configuration
changes. I was using the generate flowfiles processor to generate 100MB
flowfiles and writing as fast as possible with a PutFile processor.
The SSD posted the following on the AS-SSD benchmark (completely unoptimized):
1.8GB/sec for sequential write;
2.3GB/sec for 4K random write (64 treads);
114MB/sec for 4K random write (1 thread)
On the PCie bus, you should easily surpass 800 MB/sec, especially if your
flowfiles are large and you have few provenance events. The theoretical
bandwidth is 985 MB/sec/lane, up to 16 lanes; I was running x4. The NVMe
standard should also help with smaller flowfiles.
Hope this helps,
> On Oct 17, 2016, at 12:06 PM, Ali Nazemian <alinazem...@gmail.com> wrote:
> Dear Nifi users/developers,
> I was wondering how can I calculate the theoretical throughput of a Nifi
> server? let's suppose we can eliminate different bottlenecks such as the file
> flow rep and provenance repo bottleneck by using a very high-end SSD.
> Moreover, assume that a very high-end network infrastructure is available. In
> this case, is it possible to reach 800MB throughput per second per each
> server? Suppose each server comes with 24 disk slots. 16 disk slots are used
> for creating 8 x RAID1(SAS 10k) mount points and are dedicated to the content
> repo. Let's say each content repo can achieve 100 MB throughput. May I say
> the total throughput per each server can be 8x100=800MBps? Is it possible to
> reach this amount of throughput practically?
> Thank you very much.
> Best regards,