Hi Adrian,
There are a few things to note. First of all: yes - there is potential for
threading. However, a few things are to be considered:
Running a plugin and generating RRD file involve a few tasks. Have you
figured out in detail, where you burn your CPU cycles or your time?
Basically, there are the following candidates:
o nfdump filter: As you mentioned, using lists instead of or chains speeds
up filtering, It depends on the number of term you have but, if you have
several hundreds to thousands and more, list improve dramatically the speed
of filtering. rbtrees are much faster and a big win!
o threading of filtering: I'm currently working on nfdump-1.6.8, which should
be the final 1.6.x version, preparing the basis for nfdump-1.7.x which will
implement threading in the collector as well as in nfdump for data processing.
There will be definitely a speed gain, but nfdump-1.7.x will take some more
months to materialise.
o I/O at some point I/O is more critical than CPU, is you have thin I/O
bandwidth.
Make sure to have a good performing I/O storage.
o concurrent NfSen cycle processing: NfSen also lakes of makeing use of multiple
CPUs while processing the 5min interval. This should also be available in one
of NfSen's next release. A prototype is already running. I need some more
testing for this.
o RRD updates: Updating RRDs is a very costly process. If you have lots of
profiles,
this may take a considerable amount of time of the 5min slot. Depending on the
amount of RRDs you have to update in your plugin, consider also this factor.
Making a long story short: Yes - is some room for improvement, but things become
more complex. If you have some ideas or hint, what you think, would make sense,
feel free to share them.
Regards
- Peter
On 9/8/12 9:18 AM, Adrian Popa wrote:
> Hello everyone,
>
> I have a backend plugin that uses nfcapd with a bunch of dynamic filters to
> generate some rrds that I want (mostly traffic per prefix (input and
> output) and top AS (input and output)). So far, it has run for 2-3 years
> without major problems, but the volume of data to be searched has increased
> and now, the plugin regularly takes over 300s to finish what it has to do
> (currently it peaks at 450s).
>
> I am thinking on how to improve performance, and I can tune the some of the
> queries (use lists [12 23 34] instead of "or": dst as 12 or dst as 23 or
> dst as 34), but I think it will only bring a small speed increase. The real
> problem is my code runs nfdump with a set of parameters and then parses the
> output. Starting an external process is time (and resource) consuming, but
> I guess there's no other way around it.
>
> Instead, I plan on benefiting from my multi-core/high memory system and run
> the queries in parallel, by using threads inside the module.
>
> Before I embark in an epic journey of code rewriting, I'd like to ask if I
> can expect speed increases from this approach, or if the threads will be
> limited to a single cpu/work queue by how the plugin architecture is
> implemented.
>
> Do you have other suggestions on how to speed things up?
>
> Thanks,
> Adrian
>
>
>
> ------------------------------------------------------------------------------
> Live Security Virtual Conference
> Exclusive live event will cover all the ways today's security and
> threat landscape has changed and how IT managers can respond. Discussions
> will include endpoint security, mobile security and the latest in malware
> threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
>
>
>
> _______________________________________________
> Nfsen-discuss mailing list
> Nfsen-discuss@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/nfsen-discuss
>
--
Be nice to your netflow data. Use NfSen and nfdump :)
------------------------------------------------------------------------------
Live Security Virtual Conference
Exclusive live event will cover all the ways today's security and
threat landscape has changed and how IT managers can respond. Discussions
will include endpoint security, mobile security and the latest in malware
threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
_______________________________________________
Nfsen-discuss mailing list
Nfsen-discuss@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfsen-discuss
