I was wondering, did you use PCIe for file flow repo or provenance repo or
content repo? or all of them?
The ETL is not very complicated ETL, so do you think isn't it possible to
reach 800MBps in production even if I use PCIe for file flow repo? Is it
worth spending money on PCIe for the file flow repo?
On Tue, Oct 18, 2016 at 2:36 AM, Joe Witt <joe.w...@gmail.com> wrote:
> Thanks Lee. Your response was awesome and really made me want to get
> hands on a set of boxes like this so we could do some testing.
> On Mon, Oct 17, 2016 at 11:32 AM, Lee Laim <lee.l...@gmail.com> wrote:
> > Joe,
> > Good points regarding throughput on real flows and sustained basis. My
> > was only pushing one aspect of the system.
> > That said, I would be interested discussing/developing a more
> > test flow to capture more real world use cases. I'll check to see if that
> > conversation has started.
> > Thanks,
> > Lee
> > Lee Laim
> > 610-864-1657
> > On Oct 17, 2016, at 9:55 PM, Ali Nazemian <alinazem...@gmail.com> wrote:
> > Dear Joe,
> > Thank you very much.
> > Best regards
> > On Mon, Oct 17, 2016 at 10:08 PM, Joe Witt <joe.w...@gmail.com> wrote:
> >> Ali
> >> I suspect bottlenecks in the software itself and the flow design will
> >> become a factor before you 800 MB/s. You'd likely hit CPU efficiency
> >> issues before this caused by the flow processors themselves and due to
> >> garbage collection. Probably the most important factor though will be
> >> the transaction rate and whether the flow is configured to tradeoff
> >> some latency for higher throughput. So many variables at play but
> >> under idealized conditions and a system like you describe it is
> >> theoretically feasible to hit that value.
> >> Practically speaking I think you'd be looking at a couple hundred MB/s
> >> per server like this on real flows on a sustained basis.
> >> Thanks
> >> Joe
> >> On Sun, Oct 16, 2016 at 11:06 PM, Ali Nazemian <alinazem...@gmail.com>
> >> wrote:
> >> > Dear Nifi users/developers,
> >> > Hi,
> >> >
> >> > I was wondering how can I calculate the theoretical throughput of a
> >> > server? let's suppose we can eliminate different bottlenecks such as
> >> > file flow rep and provenance repo bottleneck by using a very high-end
> >> > SSD.
> >> > Moreover, assume that a very high-end network infrastructure is
> >> > available.
> >> > In this case, is it possible to reach 800MB throughput per second per
> >> > each
> >> > server? Suppose each server comes with 24 disk slots. 16 disk slots
> >> > used
> >> > for creating 8 x RAID1(SAS 10k) mount points and are dedicated to the
> >> > content repo. Let's say each content repo can achieve 100 MB
> >> > May
> >> > I say the total throughput per each server can be 8x100=800MBps? Is
> >> > possible to reach this amount of throughput practically?
> >> > Thank you very much.
> >> >
> >> > Best regards,
> >> > Ali
> > --
> > A.Nazemian