Hi Ben, thanks for your answer. I guess it is holiday season or is the topic that boring? ;)
On 08/21/2013 05:02 PM, Ben Allen wrote: > the biggest scale issue that we've seen ( ~30GB/day, 30 sources sending > flow data, 15 profiles defined) is read performance. > >>From what I've seen of the nfdump code, it reads the flow data one record > at a time. On a "wide" filesystem (ie. RAID3/5/6, or multi-drive ZFS > pool), the small reads kill performance because of the amount of time it > takes the platter to spin around to read the next record. > > Rewriting nfdump to do multi-threaded IO may be the biggest performance / > scaling win. Granted, moving to SSD could also solve the problem. For the every 5 minute nfprofile runs it is only slow one time reading it from harddisk. The rest of the 5 minute flow file runs of nfprofile should your filesystem cache (aka RAM) handle (at least on Linux). For custom analysis over bigger time frames or that multiplied by a handful of users could get interesting. We have 80-100GB/day with around 30 exporters as well. Profiles are likely to increase for all kinds of views. The question is how far... Anybody had problems scaling NfSen? If yes what kind of problem. And how was it solved? Regards, Axel -- Axel Fischer Network Engineer 1&1 Internet AG Brauerstrasse 48 D-76135 Karlsruhe http://www.1und1.de (AS8560) Hauptsitz Montabaur, Amtsgericht Montabaur, HRB 6484 Vorstand: Ralph Dommermuth, Frank Einhellinger, Robert Hoffmann, Andreas Hofmann, Markus Huhn, Hans-Henning Kettler, Uwe Lamnek, Jan Oetjen, Christian Würst Aufsichtsratsvorsitzender: Michael Scheeren Member of United Internet Diese E-Mail kann vertrauliche und/oder gesetzlich geschützte Informationen enthalten. Wenn Sie nicht der bestimmungsgemäße Adressat sind oder diese E-Mail irrtümlich erhalten haben, unterrichten Sie bitte den Absender und vernichten Sie diese Email. Anderen als dem bestimmungsgemäßen Adressaten ist untersagt, diese E-Mail zu speichern, weiterzuleiten oder ihren Inhalt auf welche Weise auch immer zu verwenden. This E-Mail may contain confidential and/or privileged information. If you are not the intended recipient of this E-Mail, you are hereby notified that saving, distribution or use of the content of this E-Mail in any way is prohibited. If you have received this E-Mail in error, please notify the sender and delete the E-Mail. ------------------------------------------------------------------------------ Introducing Performance Central, a new site from SourceForge and AppDynamics. Performance Central is your source for news, insights, analysis and resources for efficient Application Performance Management. Visit us today! http://pubads.g.doubleclick.net/gampad/clk?id=48897511&iu=/4140/ostg.clktrk _______________________________________________ Nfsen-discuss mailing list Nfsen-discuss@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/nfsen-discuss
