Hi Paul, (CCd to the flow-tools list as this may be useful to others and also in hope someone has a better solution!)
I don't have a direct conversion method. But I do have a workable solution that works for processing my volume of flows (approx 1.2billion/day) on a single mid-low spec server (2 x dual core Xeon CPU, SAS 3G disks). Essentially I just use a rate-limited flow-send to resend the flows into a locally listening rwflowpack. You could certainly increase the rate if you were using a pair of servers - the main thing you need to watch out for is network UDP buffer overruns causing recv errors. Parallelising with multiple senders and receivers is a natural extension if you have a large dataset and a few spare servers. Each collector would receive to local files and resend to a central Silk datastore with rwsender/rwreceiver, for example, in a fan-in arrangement. The machines are all set to have an 8Meg UDP buffer rather than the default which I think is 128K on linux with something like the following sysctl setting: # Set UDP buffer to 8M instead of the default 128K net.core.rmem_max = 8388608 The process is as follows: 0) Optionally check file validity with a flow-cat to /dev/null and check the return value and stderr. This has the side effect of caching most of the file content in the 1) Take note of current UDP RcvbufErrors SndbufErrors in /proc/net/snmp 2) Execute something like the following. Tweak the inter-packet delay (the -x option) to match your local server and network processing capacity: flow-send -V 5 -x 200 0/127.0.0.1/18001 < $file 3) Check UDP error counters again and bleat if they have increased during the operation 4) Rinse, repeat Hope this helps, Andrew ________________________________________ From: Paul King [[email protected]] Sent: 18 February 2011 23:37 To: Andrew O'Brien Subject: [Flow-tools] RE: Speeding up the flow-cat process Hi Andrew, I have found a few posts where you were asking about going from 'flow-tools' format to SiLK format. Did you ever solve that? I have a large historical store of 'flow-tools' netflow and I want to take a look at it with some tools that support SiLK format and so I need to convert. Did you find a reliable way to do this? Thanks very much Paul _______________________________________________ Flow-tools mailing list [email protected] http://mailman.splintered.net/mailman/listinfo/flow-tools
