It depends. You can still process your data faster if you are not
interested in EVERYTHING at once. You could save what you're interested in
every 5 minutes (via a plugin) to an external database and build more
elaborate queries in the database. This way you won't have to plow through
100s of gigabytes of data when you want to extract only something small.
For instance, you could query total traffic every 5 minutes with "in if 5
and dst as 20" (which would be the data of interest for you), and save this
total someplace else. You could then use the data faster in other queries.
The possibilities are limitless, depends only what you want to do.
On Mon, Jan 14, 2013 at 5:20 PM, Jason Lixfeld <
jason-nfsen-disc...@lixfeld.ca> wrote:
> Yikes. So processing a months worth of data could take hours and hours
> and hours. Sounds like it might not be the best tool for traffic
> engineering type business where large data sets would need to get crunched?
>
> On 2013-01-14, at 10:16 AM, Adrian Popa <adrian.popa...@gmail.com> wrote:
>
> > Reports are run against the collected data (nfdump files). The larger
> the time interval, the more files nfdump has to process. For my system, I
> collect about 4 days worth of netflow data, and they take up ~170GB of disk
> space. If I run a top x report on the whole interval, it will take up to an
> hour to process.
> >
> > Sadly, during this time nfdump is not returning any data (since it is
> still chewing on the problem), so your browser/firewall/proxy might
> conclude that the host is not responding and reset the session.
> >
> >
> > On Mon, Jan 14, 2013 at 5:04 PM, Jason Lixfeld <
> jason-nfsen-disc...@lixfeld.ca> wrote:
> >
> > On 2013-01-10, at 3:08 PM, Peter Haag <ph...@users.sourceforge.net>
> wrote:
> >
> > > On 10/1/13 4:40 PM, Jason Lixfeld wrote:
> > >> While trying to chew through anything more than a few days worth of
> data, I'm faced with a 504 Gateway Time-out error. Is that something that
> can be fixed within Nfsen or is that a web server thing?
> > >
> > > It's most likely a php timeout thing. If you do such overlength
> queries over the web interface, it might be a good idea
> > > to increase this time in php.ini.
> >
> > Indeed. FastCGI somethingorother.
> >
> > So it takes 4 minutes to process about a weeks worth of data meaning it
> could take about 15-20 minutes to process a months worth of data. This
> seems like a long period of time to me. Is this normal or what factors
> would affect nfdump's timeliness in processing files?
> >
> ------------------------------------------------------------------------------
> > Master Visual Studio, SharePoint, SQL, ASP.NET, C# 2012, HTML5, CSS,
> > MVC, Windows 8 Apps, JavaScript and much more. Keep your skills current
> > with LearnDevNow - 3,200 step-by-step video tutorials by Microsoft
> > MVPs and experts. SALE $99.99 this month only -- learn more at:
> > http://p.sf.net/sfu/learnmore_122412
> > _______________________________________________
> > Nfsen-discuss mailing list
> > Nfsen-discuss@lists.sourceforge.net
> > https://lists.sourceforge.net/lists/listinfo/nfsen-discuss
> >
>
>
------------------------------------------------------------------------------
Master Visual Studio, SharePoint, SQL, ASP.NET, C# 2012, HTML5, CSS,
MVC, Windows 8 Apps, JavaScript and much more. Keep your skills current
with LearnDevNow - 3,200 step-by-step video tutorials by Microsoft
MVPs and experts. SALE $99.99 this month only -- learn more at:
http://p.sf.net/sfu/learnmore_122412
_______________________________________________
Nfsen-discuss mailing list
Nfsen-discuss@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfsen-discuss
