Welp, I'm just a dumb router jockey, so perl, databases and shell scripts are a little beyond me, I'm afraid :(
On 2013-01-14, at 3:18 PM, Adrian Popa <adrian.popa...@gmail.com> wrote: > Well, I don't know if there is a specific plugin that does this already, but > with a little perl knowledge you can write your own plugin. Otherwise, you > can achieve the same result with cron and shell scripts to collect and parse > the data... > > But it's like a DIY kit - some assembly required :) > > > On Mon, Jan 14, 2013 at 10:05 PM, Jason Lixfeld > <jason-nfsen-disc...@lixfeld.ca> wrote: > Which plugin would do that? I'd be interested in saving AS info so at the > end of the month I could see what the top 100 AS we exchange traffic with, > possibly bgp-next-hop as well. > > On 2013-01-14, at 10:31 AM, Adrian Popa <adrian.popa...@gmail.com> wrote: > > > It depends. You can still process your data faster if you are not > > interested in EVERYTHING at once. You could save what you're interested in > > every 5 minutes (via a plugin) to an external database and build more > > elaborate queries in the database. This way you won't have to plow through > > 100s of gigabytes of data when you want to extract only something small. > > > > For instance, you could query total traffic every 5 minutes with "in if 5 > > and dst as 20" (which would be the data of interest for you), and save this > > total someplace else. You could then use the data faster in other queries. > > > > The possibilities are limitless, depends only what you want to do. > > > > > > On Mon, Jan 14, 2013 at 5:20 PM, Jason Lixfeld > > <jason-nfsen-disc...@lixfeld.ca> wrote: > > Yikes. So processing a months worth of data could take hours and hours and > > hours. Sounds like it might not be the best tool for traffic engineering > > type business where large data sets would need to get crunched? > > > > On 2013-01-14, at 10:16 AM, Adrian Popa <adrian.popa...@gmail.com> wrote: > > > > > Reports are run against the collected data (nfdump files). The larger the > > > time interval, the more files nfdump has to process. For my system, I > > > collect about 4 days worth of netflow data, and they take up ~170GB of > > > disk space. If I run a top x report on the whole interval, it will take > > > up to an hour to process. > > > > > > Sadly, during this time nfdump is not returning any data (since it is > > > still chewing on the problem), so your browser/firewall/proxy might > > > conclude that the host is not responding and reset the session. > > > > > > > > > On Mon, Jan 14, 2013 at 5:04 PM, Jason Lixfeld > > > <jason-nfsen-disc...@lixfeld.ca> wrote: > > > > > > On 2013-01-10, at 3:08 PM, Peter Haag <ph...@users.sourceforge.net> wrote: > > > > > > > On 10/1/13 4:40 PM, Jason Lixfeld wrote: > > > >> While trying to chew through anything more than a few days worth of > > > >> data, I'm faced with a 504 Gateway Time-out error. Is that something > > > >> that can be fixed within Nfsen or is that a web server thing? > > > > > > > > It's most likely a php timeout thing. If you do such overlength queries > > > > over the web interface, it might be a good idea > > > > to increase this time in php.ini. > > > > > > Indeed. FastCGI somethingorother. > > > > > > So it takes 4 minutes to process about a weeks worth of data meaning it > > > could take about 15-20 minutes to process a months worth of data. This > > > seems like a long period of time to me. Is this normal or what factors > > > would affect nfdump's timeliness in processing files? > > > ------------------------------------------------------------------------------ > > > Master Visual Studio, SharePoint, SQL, ASP.NET, C# 2012, HTML5, CSS, > > > MVC, Windows 8 Apps, JavaScript and much more. Keep your skills current > > > with LearnDevNow - 3,200 step-by-step video tutorials by Microsoft > > > MVPs and experts. SALE $99.99 this month only -- learn more at: > > > http://p.sf.net/sfu/learnmore_122412 > > > _______________________________________________ > > > Nfsen-discuss mailing list > > > Nfsen-discuss@lists.sourceforge.net > > > https://lists.sourceforge.net/lists/listinfo/nfsen-discuss > > > > > > > > > ------------------------------------------------------------------------------ Master Visual Studio, SharePoint, SQL, ASP.NET, C# 2012, HTML5, CSS, MVC, Windows 8 Apps, JavaScript and much more. Keep your skills current with LearnDevNow - 3,200 step-by-step video tutorials by Microsoft MVPs and experts. SALE $99.99 this month only -- learn more at: http://p.sf.net/sfu/learnmore_122412 _______________________________________________ Nfsen-discuss mailing list Nfsen-discuss@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/nfsen-discuss
