Although the data structure of nfcapd.current is the same, it can not be read
from, while a collector still adds data to this file. You would end up in data
corruption. Therefore a flag prohibits data reading.
The error you get is not because of compression. Compression is fully
transparent
you do not need to care about that. The error results from piping files with
cat, which does not work because of the file structure as of nfdump.
If you need to read among multiple files you simple do:
./nfdump -R nfcapd.201411141000:nfcapd.201411141005
which means -R <first:last> which reads everything from first up to and
including last
file. This may span way more files than cat can and does not need another
process.
So the only restriction you have is - no read from nfcapd.current
Regards
- Peter
On 14.11.14 17:48, Logan Vig wrote:
> What I need to do is nfdump from both the last 5 minute rolled file
> (currently nfcapd.201411141005) and nfcapd.current.18967. Currently my
> scripts simply use the latest rolled nfcapd file which introduces up
> to 5 minutes of latency. For my purposes I need to analyze netflow
> data as real time as possible with nfdump.
>
> In theory this should be as simple as loading both nfcapd.201411141005
> and nfcapd.current.18967 into nfdump at the same time as one would
> with a range of similarly named files using -R or a number of other
> options available in nfdump. Unfortunately in practice I have not
> found a way to load both of these files simultaneously for analysis in
> the same nfdump.
>
> Here are some methods I have tried.
>
> Specifying multiple -r opts on the command line like:
> nfdump -r nfcapd.201411141005 -r nfcapd.current.18967 -n 1 -s
> record/bps -A proto,dstip -o extended
> result: only reads 1 file, the last -r option, in this case
> nfcapd.current.18967
>
> Using stdin to read multiple files:
> cat nfcapd.201411141000 nfcapd.201411141005 | nfdump -n 1 -s
> record/bps -A proto,dstip -o extended
> Results in:
>
> ReadBlock() error decompression failed in nffile.c line 779: LZO error: -4
> Skip corrupt data file '(null)'
>
> So the main problem I am having here is piping multiple files to
> nfdump via stdin. I receive the above error any time I attempt to pipe
> multiple files to nfdump. Since the error was LZO related, I tried
> decompressing both files first..
>
> # nfdump -j nfcapd.201411140930
> Uncompress file nfcapd.201411140930 ..
> # nfdump -j nfcapd.201411140935
> Uncompress file nfcapd.201411140935 ..
> # cat nfcapd.20141114093* | nfdump -n 1 -s record/bps -A proto,dstip -o
> extended
> Can't process block type 670. Skip block.
> Can't process block type 0. Skip block.
> Corrupt data file: Requested buffer size 3489660928 exceeds max. buffer size.
> Skip corrupt data file '(null)'
>
> Reading the files separately works fine and yields no errors. Any
> ideas on how I can get nfdump to read nfcapd.201411141005 and
> nfcapd.current.18967 in one run would be appreciated.
>
> ------------------------------------------------------------------------------
> Comprehensive Server Monitoring with Site24x7.
> Monitor 10 servers for $9/Month.
> Get alerted through email, SMS, voice calls or mobile push notifications.
> Take corrective actions from your mobile device.
> http://pubads.g.doubleclick.net/gampad/clk?id=154624111&iu=/4140/ostg.clktrk
> _______________________________________________
> Nfdump-discuss mailing list
> Nfdump-discuss@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/nfdump-discuss
>
--
Be nice to your netflow data. Use NfSen and nfdump :)
------------------------------------------------------------------------------
Comprehensive Server Monitoring with Site24x7.
Monitor 10 servers for $9/Month.
Get alerted through email, SMS, voice calls or mobile push notifications.
Take corrective actions from your mobile device.
http://pubads.g.doubleclick.net/gampad/clk?id=154624111&iu=/4140/ostg.clktrk
_______________________________________________
Nfdump-discuss mailing list
Nfdump-discuss@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfdump-discuss
