Ali, Without knowing the details of the data streams, nature of each event and the operations that will be performed against them, or how the processors themselves will work, I cannot give you a solid answer. Do I think it is possible? Absolutely. Do I think there will be hurdles to overcome to reach and sustain such a rate? Absolutely.
Thanks Joe On Mon, Oct 17, 2016 at 9:28 PM, Lee Laim <lee.l...@gmail.com> wrote: > Ali, > I used the pcie for all repos and the PutFile destination. > > > > On Oct 18, 2016, at 8:38 AM, Ali Nazemian <alinazem...@gmail.com> wrote: > > Hi Lee, > > I was wondering, did you use PCIe for file flow repo or provenance repo or > content repo? or all of them? > > Joe, > > The ETL is not very complicated ETL, so do you think isn't it possible to > reach 800MBps in production even if I use PCIe for file flow repo? Is it > worth spending money on PCIe for the file flow repo? > > Best regards > > On Tue, Oct 18, 2016 at 2:36 AM, Joe Witt <joe.w...@gmail.com> wrote: >> >> Thanks Lee. Your response was awesome and really made me want to get >> hands on a set of boxes like this so we could do some testing. >> >> Thanks >> Joe >> >> On Mon, Oct 17, 2016 at 11:32 AM, Lee Laim <lee.l...@gmail.com> wrote: >> > Joe, >> > Good points regarding throughput on real flows and sustained basis. My >> > test >> > was only pushing one aspect of the system. >> > >> > That said, I would be interested discussing/developing a more >> > comprehensive >> > test flow to capture more real world use cases. I'll check to see if >> > that >> > conversation has started. >> > >> > Thanks, >> > Lee >> > >> > >> > >> > >> > >> > Lee Laim >> > 610-864-1657 >> > >> > On Oct 17, 2016, at 9:55 PM, Ali Nazemian <alinazem...@gmail.com> wrote: >> > >> > Dear Joe, >> > Thank you very much. >> > >> > Best regards >> > >> > >> > On Mon, Oct 17, 2016 at 10:08 PM, Joe Witt <joe.w...@gmail.com> wrote: >> >> >> >> Ali >> >> >> >> I suspect bottlenecks in the software itself and the flow design will >> >> become a factor before you 800 MB/s. You'd likely hit CPU efficiency >> >> issues before this caused by the flow processors themselves and due to >> >> garbage collection. Probably the most important factor though will be >> >> the transaction rate and whether the flow is configured to tradeoff >> >> some latency for higher throughput. So many variables at play but >> >> under idealized conditions and a system like you describe it is >> >> theoretically feasible to hit that value. >> >> >> >> Practically speaking I think you'd be looking at a couple hundred MB/s >> >> per server like this on real flows on a sustained basis. >> >> >> >> Thanks >> >> Joe >> >> >> >> On Sun, Oct 16, 2016 at 11:06 PM, Ali Nazemian <alinazem...@gmail.com> >> >> wrote: >> >> > Dear Nifi users/developers, >> >> > Hi, >> >> > >> >> > I was wondering how can I calculate the theoretical throughput of a >> >> > Nifi >> >> > server? let's suppose we can eliminate different bottlenecks such as >> >> > the >> >> > file flow rep and provenance repo bottleneck by using a very high-end >> >> > SSD. >> >> > Moreover, assume that a very high-end network infrastructure is >> >> > available. >> >> > In this case, is it possible to reach 800MB throughput per second per >> >> > each >> >> > server? Suppose each server comes with 24 disk slots. 16 disk slots >> >> > are >> >> > used >> >> > for creating 8 x RAID1(SAS 10k) mount points and are dedicated to the >> >> > content repo. Let's say each content repo can achieve 100 MB >> >> > throughput. >> >> > May >> >> > I say the total throughput per each server can be 8x100=800MBps? Is >> >> > it >> >> > possible to reach this amount of throughput practically? >> >> > Thank you very much. >> >> > >> >> > Best regards, >> >> > Ali >> > >> > >> > >> > >> > -- >> > A.Nazemian > > > > > -- > A.Nazemian
