Axel,
the biggest scale issue that we've seen ( ~30GB/day, 30 sources sending
flow data, 15 profiles defined) is read performance.
>From what I've seen of the nfdump code, it reads the flow data one record
at a time. On a "wide" filesystem (ie. RAID3/5/6, or multi-drive ZFS
pool), the small reads kill performance because of the amount of time it
takes the platter to spin around to read the next record.
Rewriting nfdump to do multi-threaded IO may be the biggest performance /
scaling win. Granted, moving to SSD could also solve the problem.
On Thu, Aug 15, 2013 at 12:17 PM, Axel Fischer <axel.fisc...@1und1.de>wrote:
> Hi there,
> beside Peters presentation at a CESNET event
> (
> http://archiv.ces.net/events/2012/campus-monitoring/p/haag-nfdump-nfsen.pdf
> )
> and a few side notes in short threads on this mailinglist I didn't find a
> lot on
> this topic.
> Collecting the flow data and writing it to different files does scale to a
> higher level than three digit profiles or even more do.
> What are your experiences? How far did you get till you needed to split
> exporters over different nfsen servers? Are hundreds or thousands of
> profiles
> possible without the need for a mainframe? And what about alerts?
>
> Or did I miss something obvious?
>
> @Peter: The roadmap for NfSen 2.0 looks nice. Is there any chance that you
> give
> us a rough estimate when there could be a release? Do you need testers or
> feedback?
>
>
> Regards,
> Axel
>
> --
> Axel Fischer
> Network Engineer
>
> 1&1 Internet AG
> Brauerstrasse 48
> D-76135 Karlsruhe
>
> http://www.1und1.de (AS8560)
>
> Hauptsitz Montabaur, Amtsgericht Montabaur, HRB 6484
>
> Vorstand: Ralph Dommermuth, Frank Einhellinger, Robert Hoffmann,
> Andreas Hofmann, Markus Huhn, Hans-Henning Kettler, Uwe Lamnek,
> Jan Oetjen, Christian Würst
> Aufsichtsratsvorsitzender: Michael Scheeren
>
> Member of United Internet
>
> Diese E-Mail kann vertrauliche und/oder gesetzlich geschützte Informationen
> enthalten. Wenn Sie nicht der bestimmungsgemäße Adressat sind oder diese
> E-Mail
> irrtümlich erhalten haben, unterrichten Sie bitte den Absender und
> vernichten
> Sie diese Email. Anderen als dem bestimmungsgemäßen Adressaten ist
> untersagt,
> diese E-Mail zu speichern, weiterzuleiten oder ihren Inhalt auf welche
> Weise
> auch immer zu verwenden.
>
> This E-Mail may contain confidential and/or privileged information. If you
> are
> not the intended recipient of this E-Mail, you are hereby notified that
> saving,
> distribution or use of the content of this E-Mail in any way is
> prohibited. If
> you have received this E-Mail in error, please notify the sender and
> delete the
> E-Mail.
>
>
> ------------------------------------------------------------------------------
> Get 100% visibility into Java/.NET code with AppDynamics Lite!
> It's a free troubleshooting tool designed for production.
> Get down to code-level detail for bottlenecks, with <2% overhead.
> Download for free and get started troubleshooting in minutes.
> http://pubads.g.doubleclick.net/gampad/clk?id=48897031&iu=/4140/ostg.clktrk
> _______________________________________________
> Nfsen-discuss mailing list
> Nfsen-discuss@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/nfsen-discuss
>
------------------------------------------------------------------------------
Introducing Performance Central, a new site from SourceForge and
AppDynamics. Performance Central is your source for news, insights,
analysis and resources for efficient Application Performance Management.
Visit us today!
http://pubads.g.doubleclick.net/gampad/clk?id=48897511&iu=/4140/ostg.clktrk
_______________________________________________
Nfsen-discuss mailing list
Nfsen-discuss@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfsen-discuss
