Ryan, I use a wrapper written i bash that allows for cluster processing
through GNU Parallel. Nfdump together with parallel are installed on
several machines which all have read access to the same repository of
nfcapd files. The wrapper splits the query into multiple jobs, e.g. split
on source router or timeslots, before instructing the cluster to process
the individual jobs on the different cluster nodes. By using this simple
program I've gained a significant performance increase.
- Tor
On Mon, May 20, 2013 at 5:48 PM, Ryan Harden <harde...@uchicago.edu> wrote:
> Has anyone written a method for distributing nfdump query processing to a
> compute cluster?
> We have a compute cluster (condor based) that I'd like to utilize when
> processing nfdump data.
>
> >From what I understand, some sort of wrapper would be required that would
> instruct N number of compute nodes to map certain data files, process them,
> return values, then another process would aggregate the values based on the
> original request and generate the output the user was looking for.
>
> Perhaps I'm thinking about this all wrong?
>
> Thoughts? Comments?
>
> /Ryan
>
> Ryan Harden
> Senior Network Engineer
> University of Chicago - AS160
> P: 773-834-5441
>
>
>
>
>
>
> ------------------------------------------------------------------------------
> AlienVault Unified Security Management (USM) platform delivers complete
> security visibility with the essential security capabilities. Easily and
> efficiently configure, manage, and operate all of your security controls
> from a single console and one unified framework. Download a free trial.
> http://p.sf.net/sfu/alienvault_d2d
> _______________________________________________
> Nfdump-discuss mailing list
> Nfdump-discuss@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/nfdump-discuss
>
------------------------------------------------------------------------------
AlienVault Unified Security Management (USM) platform delivers complete
security visibility with the essential security capabilities. Easily and
efficiently configure, manage, and operate all of your security controls
from a single console and one unified framework. Download a free trial.
http://p.sf.net/sfu/alienvault_d2d
_______________________________________________
Nfdump-discuss mailing list
Nfdump-discuss@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfdump-discuss
