Thanks Miro - it should go in 1.6.14
- Peter
On 06/09/15 17:32, Miroslav Kratochvil wrote:
> Hello,
>
> just finished the implementation.
>
> I kindof found the Peter's github repository, so I forked&initialized it up
> with latest stable version, converted the source a bit with styling tools
> (sorry for that if there already was some other coding culture -- still, the
> patch in question is
> pretty easy to rip out and apply by hand even to the previous code) and added
> BZ2 support (with -J option using libbz2, for similarity with tar, and
> because -Z was already taken).
>
> Links:
>
> my repo:
> https://github.com/exaexa/nfdump
>
> the implementation diff:
> https://github.com/exaexa/nfdump/commit/e6198ed8bd1f4e136e339b124294013368cc3592
>
> the pull request:
> https://github.com/phaag/nfdump/pull/1
>
> Thanks for any useful comments! One simple idea that comes to my mind is to
> bundle the bz2 library with the code -- it would simplify the packaging and
> it's already the same case as with minilzo... Also, I just tested it and it
> works, but more
> testers would certainly help it (I'm not sure I was able to test all the 4
> bigger chunks of code that I put there, e.g. what exactly is WriteExtraBlock
> good for?).
>
> Cheers,
> -mk
>
>
>
> PS. I just noticed that the SF mailing list somehow ignores my e-mails, so
> the discussion in the list is going to look a bit messy. I hope everyone will
> be able to sort it out. :]
>
> On Sun, 6 Sep 2015 at 16:31 Michael Hare <michael.h...@wisc.edu
> <mailto:michael.h...@wisc.edu>> wrote:
>
> I also would appreciate the option. For us storage is more expensive
> than processing.
>
> -Michael
>
> > -----Original Message-----
> > From: Peter Haag [mailto:ph...@users.sourceforge.net
> <mailto:ph...@users.sourceforge.net>]
> > Sent: Sunday, September 06, 2015 6:35 AM
> > To: Miroslav Kratochvil <exa....@gmail.com <mailto:exa....@gmail.com>>;
> nfdump-
> > disc...@lists.sourceforge.net <mailto:disc...@lists.sourceforge.net>
> > Subject: Re: [Nfdump-discuss] Compression possibilities of nfdump (and
> one
> > weird issue)
> >
> > Hi,
> > the compress algorithm was used as best compromise between speed,
> > result and CPU usage.
> > I also did test bzip2 and - yes - you get more out of disk space, but
> it's more
> > CPU intensive.
> >
> > Anyway, it would be possible to add another compression along the
> existing
> > one and let the user choose.
> > So if this is a demand from other people too, we can happily have a
> lookk
> > together.
> >
> > Cheers
> >
> > - Peter
> >
> >
> > On 05/09/15 16:01, Miroslav Kratochvil wrote:
> > > Hello list,
> > >
> > > I was wondering whether it is possible to use any better compression
> for
> > nfdump files than the supplied LZO1X-1. By simple measurements at my
> site,
> > the files compressed by the nfdump are shrank to around 38% of the
> original
> > size, while using bzip2
> > > (speed of which is still bearable for many environments) gets to
> around
> > 20% of the original.
> > >
> > > For people who need to archive the netflow data, almost two times
> smaller
> > archives could be a great space/time saver. CPU usage of the
> compression is
> > IMHO not a real issue on current machines.
> > >
> > > Is there any possibility to replace the LZO compression by something
> else?
> > I'd happily code a patch for bzip2 support -- is that a viable option,
> or is there
> > some reason not to use this specific compression? (I've seen github
> > migration is underway,
> > > I'd just send a merge request if it'd been finished :] )
> > >
> > > Thanks a lot in advance,
> > > -mk
> > >
> > >
> > > PS.:
> > > Also, on a related note, when trying to code the compression around,
> I've
> > seen this weird error unexplainable by traditional means:
> > >
> > > $ cat < somefile |nfdump -r -
> > > [... data are OK]
> > >
> > > $ bzip2 somefile
> > > $ bzip2 -c -d < somefile.bz2 |nfdump -r -
> > > Date flow start Duration Pr[....]
> > > Read error in file '(null)': Success
> > > No matched flows
> > >
> > > Any explanation?
> > >
> > >
> >
> >
> ------------------------------------------------------------------------------
> > _______________________________________________
> > Nfdump-discuss mailing list
> > Nfdump-discuss@lists.sourceforge.net
> <mailto:Nfdump-discuss@lists.sourceforge.net>
> > https://lists.sourceforge.net/lists/listinfo/nfdump-discuss
>
------------------------------------------------------------------------------
_______________________________________________
Nfdump-discuss mailing list
Nfdump-discuss@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfdump-discuss
