Thanks Peter. I guess I did not read through the nfsen-dist.conf file as I used my config from a much older version durring my upgrade. The only thing it complianed about during the install (after the "-" issue from the other thread) was the high water mark format. Anyway moving along.
Another question along these lines. I have about 1.2 million files on average, so if my compression tests are satisfactory I will want to convert them. By running the manual compression on them, do you know what happens if I run a big recursive script to compress them and it encounters a file that is already compressed or do I need to validate each file first by parsing the output of nfdump -v? I belive the documentation suggests that the "-j" option is bi-directional, meaning it will compress uncompressed files and decompress compressed ones. Just looking for clarification. Also, if I change (or in my case add) the compression parameters do I need to do a reconfig? Thanks! Chris Waters Technology Services - Network Group JELD-WEN, Inc. Information Systems [EMAIL PROTECTED] -----Original Message----- From: Peter Haag [mailto:[EMAIL PROTECTED] Sent: Tuesday, March 11, 2008 12:53 AM To: Chris Waters; [email protected] Subject: Re: [Nfsen-discuss] Compression in 1.5.7 -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Hi Chris, Compression is configured in nfsen.conf: # Compress flows while collecting 0 or 1 $ZIPcollected = 1; # Compress flows in profiles 0 or 1 $ZIPprofiles = 1; Which means the collected flows and/or the created profiles are compressed. These tags should be in nfsen-dist.conf with comes with NfSen. As of nfcapd itself - it compresses the flows be adding -z as described in nfcapd(1). Whether you want to compress your flows or not, depends on your system I/O speed , CPU or disk capacity. You can do a rather small test by running. ./nftest <path/to/a/nfcapd.file> ( nftest is only in the src directory and built, when doing a 'make test' ) It does not alter the file but takes your flow data to compare speed and compression. The compression algorithm selected is a bit less effective than zip, but *very* fast, what matters most, when processing flows. As the size of the files shrinks down to 50% in average, I think this is a good trade-off. Already existing nfcapd files can be compressed/uncompressed by using ./nfdump -j nfcapd.file Hope this helps. - Peter - --On March 10, 2008 14:27:35 -0700 Chris Waters <[EMAIL PROTECTED]> wrote: | I would like to know how to use the compression feature of nfdump 1.5.7. | I'm not sure where it's documented but I don't seem to be able to find | in the mail archives or in the man pages for it. | | 1. How do you enable it? It seems that it should be an config option | in nfsen.conf but I don't see it. I am using nfdump 1.5.7 with a | current svn snapshot of nfsen. It doesn't look like compression is on | by default. | | Here is a current file: | [root]# /usr/local/svn/nfdump/bin/nfdump -v nfcapd.200803101335 | File : nfcapd.200803101335 | Version : 1 - not compressed | Blocks : 1 | Records : 15900 | | Here is a current nfcapd process: | netflow 22959 1 0 Mar09 ? 00:00:00 | /usr/local/svn/nfdump/bin/nfcapd -w -D -I rca-cal-md-1 -p 10060 -u | netflow -g apache -B 200000 -l /data/nfsen/profiles/live/rca-cal-md-1 | -P /data/nfsen/var/run/rca-cal-md-1.pid | | 2. Once enabled, what performance impact will it have when running | nfsen? Longer query times etc? | | 3. Anything else to watch out for? | | If I have missed the documentation, please just point me to it. | | Thanks. | | | Chris Waters | | Technology Services - Networks Group | | | | JELD-WEN, inc. | | Information Systems | | [EMAIL PROTECTED] | | RELIABILITY for real life(r) | | This correspondence is for the named person's use only. It may contain | confidential or legally privileged information and is intended solely | for the named addressee. If you receive this correspondence in error, | please notify the sender and delete it from your system. You must not | disclose, copy or rely on any part of this correspondence if you are | not the intended recipient. | | - -- _______ SWITCH - The Swiss Education and Research Network ______ Peter Haag, Security Engineer, Member of SWITCH CERT PGP fingerprint: D9 31 D5 83 03 95 68 BA FB 84 CA 94 AB FC 5D D7 SWITCH, Werdstrasse 2, P.O. Box, CH-8021 Zurich, Switzerland E-mail: [EMAIL PROTECTED] Web: http://www.switch.ch/ -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.3 (Darwin) iQCVAwUBR9Y6ef5AbZRALNr/AQKgywQAipn/TD5YAhVPfrDa4yQXBcvpE8H3qKxq Q+cfdQB4FxCQ5EgZBZcmaIgw3FQDxQX2ElZpxImIbnStwPtzCCrxJk72uh4b1vng H3wBTaytJOhNcwocrE3CJ1olc2rsff7WYBQggPQR5O/NAcjC8Y0N/oYMN88UBk8r Lfjq1pwU85I= =8TT8 -----END PGP SIGNATURE----- ------------------------------------------------------------------------- This SF.net email is sponsored by: Microsoft Defy all challenges. Microsoft(R) Visual Studio 2008. http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/ _______________________________________________ Nfsen-discuss mailing list [email protected] https://lists.sourceforge.net/lists/listinfo/nfsen-discuss
