I've noticed that the total# of flow files and processors is missing from
the questions. Since NIFI keeps state on disk, every transaction has to be
committed. Some newer processors support batch mode, but worst case is
absent of this. Thus the limit might not be in bytes but number of flow
file
Ali,
Without knowing the details of the data streams, nature of each event
and the operations that will be performed against them, or how the
processors themselves will work, I cannot give you a solid answer. Do
I think it is possible? Absolutely. Do I think there will be hurdles
to overcome to
Ali,
I used the pcie for all repos and the PutFile destination.
> On Oct 18, 2016, at 8:38 AM, Ali Nazemian wrote:
>
> Hi Lee,
>
> I was wondering, did you use PCIe for file flow repo or provenance repo or
> content repo? or all of them?
>
> Joe,
>
> The ETL is not very complicated ETL,
Hi Lee,
I was wondering, did you use PCIe for file flow repo or provenance repo or
content repo? or all of them?
Joe,
The ETL is not very complicated ETL, so do you think isn't it possible to
reach 800MBps in production even if I use PCIe for file flow repo? Is it
worth spending money on PCIe fo
Thanks Lee. Your response was awesome and really made me want to get
hands on a set of boxes like this so we could do some testing.
Thanks
Joe
On Mon, Oct 17, 2016 at 11:32 AM, Lee Laim wrote:
> Joe,
> Good points regarding throughput on real flows and sustained basis. My test
> was only pushi
Joe,
Good points regarding throughput on real flows and sustained basis. My test
was only pushing one aspect of the system.
That said, I would be interested discussing/developing a more comprehensive
test flow to capture more real world use cases. I'll check to see if that
conversation has sta
Hi Ali,
I observed ~1GB/sec on a test PutFile processor using an enterprise PCIe NVMe
ssd on a single instance on desktop class hardware. I plan to run more in
depth tests on server class hardware but will likely be on 1 Gb network. I
should note I'm not sure exactly how much provenance was b
Dear Joe,
Thank you very much.
Best regards
On Mon, Oct 17, 2016 at 10:08 PM, Joe Witt wrote:
> Ali
>
> I suspect bottlenecks in the software itself and the flow design will
> become a factor before you 800 MB/s. You'd likely hit CPU efficiency
> issues before this caused by the flow processor
Ali
I suspect bottlenecks in the software itself and the flow design will
become a factor before you 800 MB/s. You'd likely hit CPU efficiency
issues before this caused by the flow processors themselves and due to
garbage collection. Probably the most important factor though will be
the transacti
