2015-01-19 19:47 GMT+02:00 John P. Rouillard <rou...@cs.umb.edu>:
> In message <e1781bcc862e47809a0a8d87e18af...@wsp5042a.baltic.seb.net>,
> Risto Vaarandi writes:
>
> >I am currently working on the 2.7.7 version, and a recent e-mail
> >exchange with one of the users has inspired me to think about
> >changing default values for --bufsize and --jointbuf/--nojointbuf
> >options.
> > [...]
> >Nevertheless, when bufsize=1 would be default, some of the code for
> >the single-line case could be factored out and written more
> >efficiently, which would allow for some performance gains for
> >single-line scenario. The downside of changing the default from
> >bufsize=10 to bufsize=1 would be the need to set --bufsize explicitly
> >on command line, in order to make pattern types like RegExp3,
> >NRegExp2, and PerlFunc5 to work. So far, there has been rarely a need
> >for this, since --bufsize=10 has been sufficient for most of the
> >cases.
> >[...]
> >Also, currently SEC assumes --jointbuf option by default which means
> >that in the case of multi-line matching all events are stored into
> >the same input buffer. Nevertheless, in this case --nojointbuf would
> >make more sense, since that creates a separate buffer for each input
> >source, allowing multiline patterns to work on data from one source
> >only. Since with bufsize=1 there is no difference between --jointbuf
> >and --nojointbuf, the --nojointbuf option would be a more reasonable
> >default.
>
> I think this makes sense. What I suggest is:
>
> if --buffsize > N and any RegExpN, PerlFuncN ... multiline match rules
> are defined, SEC should throw an error ("Use --bufsize=N on command
> line, currentl setting is too small to allow this rule to work").Then
> SEC should exit. I would conssider that a fatal error. This is a
> little different from how SEC currently handles invalid rules, but I
> think this change would warrant SEC exiting.
>
Seems like a good idea, since in order to get things work, one has to
change the command line which can't be done without starting a new process.
There is one subtle exception here, though -- if SEC has been started with
the SECRC environment variable pointing to a valid resource file, command
line options can be added and modified through this file. Among other
things, once can change the --bufsize parameter, and also switch between
--jointbuf/nojointbuf modes (such runtime changes are handled by the code).
This means that if SECRC variable is set, it is worthwhile to keep the
process running and only log a relevant message with a high severity.
>
> If bufsize > 1 then --nojointbuf becomes the default (it might as well
> be the default for size=1 as well, but it's effectively a noop in that
> case so it doesn't matter). I have never had multiline matches with
> multiple inputs work reliably. There was always a race condition. I
> have always used nojointbuf with multiple input streams and multiline
> match rules.
>
> So I think those are both good ideas. One thing I will note is that a
> project I am working on supplies events in a multiline json data
> structure, so making sure that the refactoring doesn't make multiline
> process/matching worse is critical to me.
>
the only thing which I am thinking of adding to the code are few code
blocks
if ($bufsize == 1) {
}
which appear in the beginning of pattern matching functions (most notably
match_regexp() and match_perlfunc()). So the effect of these code blocks
should be minimal, since they only involve the evaluation of one extra
condition for the multiline case.
kind regards,
risto
> --
> -- rouilj
> John Rouillard
> ===========================================================================
> My employers don't acknowledge my existence much less my opinions.
>
>
> ------------------------------------------------------------------------------
> New Year. New Location. New Benefits. New Data Center in Ashburn, VA.
> GigeNET is offering a free month of service with a new server in Ashburn.
> Choose from 2 high performing configs, both with 100TB of bandwidth.
> Higher redundancy.Lower latency.Increased capacity.Completely compliant.
> http://p.sf.net/sfu/gigenet
> _______________________________________________
> Simple-evcorr-users mailing list
> Simple-evcorr-users@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/simple-evcorr-users
>
------------------------------------------------------------------------------
New Year. New Location. New Benefits. New Data Center in Ashburn, VA.
GigeNET is offering a free month of service with a new server in Ashburn.
Choose from 2 high performing configs, both with 100TB of bandwidth.
Higher redundancy.Lower latency.Increased capacity.Completely compliant.
http://p.sf.net/sfu/gigenet
_______________________________________________
Simple-evcorr-users mailing list
Simple-evcorr-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/simple-evcorr-users
