On Sun, 28 Feb 2010, Rainer Gerhards wrote:
> Quickly from the mobile: pri based filters require *excatly* one single word
> (!) lookup currently. You can't beat that.
>
> I am not totally sure how you think about parse trees. In my view, parsin and
> filteing are two different things./stages. Maybe we have different concepts
> on our minds?
I am probably using the wrong terminology here.
say you have a set of rules that say
if programname startswith abc log to /abc
if programname startswith acc log to /acc
if programname startswith acc log to @1.1.1.1
if programname startswith bcd log to /bcd
and assuming that these were the only possible programnames for simplicity
in the explination.
the filtering logic would go somthing like this
the rules would compile into a tree
-a-b -> /abc
-c -> /acc,@1.1.1.1
-b -> /bcd
receive programname abc
I have no facility/Pri filter rules
I have no time filter rules
I have no system filter rules
I have programname filter rules
look at the first character of the programname
it's "a", I have more than one rule that fits that.
look at the second character of the programname
it's a "b", There are no other decisions to make,
invoke the action /abc
receive progranmane bcd
I have no facility/PRI filter rules
I have no time filter rules
I have no system filter rules
I have programname filter rules
look at the first character of the programname
it's a "b", There are no other decisions to make,
invoke the action /bcd
receive programname acc
I have no facility/Pri filter rules
I have no time filter rules
I have no system filter rules
I have programname filter rules
look at the first character of the programname
it's "a", I have more than one rule that fits that.
look at the second character of the programname
it's a "c", There are no other decisions to make,
invoke the action /acc and the action @1.1.1.1
similarly the facility/pri filtering rules would be either compiled into a
tree, or (since it's 256 entries total) into a lookup table with pointers
to the list of actions to take for that entry
the idea is to spend the time at startup to create the tree that
represents the ruleset in order to speed up the processing of each
individual message.
the real tree would be a bit more complicated than I describe here as it
needs to have 'anything else' entries between the known entries (which
means that it would not be able to shortcut quite as much as I describe),
and have provision for 'do a more complicated thing (like regex) here and
if it matches continue'. but except for regex matches, this would make
processing the rules pretty much independant of how many rules there were
or how complex the ruleset is.
there doe snot need to be one single programname filter tree, if you had a
rule that said
if pri = info and programname startswith def then def
the pri tree/table would have an entry for info that would point to a
filter tree for programname that just had the check for def in it
am I makeing any more sense now?
David Lang
> rainer
>
> ----- Urspr?ngliche Nachricht -----
> Von: "[email protected]" <[email protected]>
> An: "rsyslog-users" <[email protected]>
> Gesendet: 28.02.10 11:28
> Betreff: Re: [rsyslog] Feedback requested: inconsistency in config statements
>
> On Sun, 28 Feb 2010, Rainer Gerhards wrote:
>
>>>
>>> if the type is depriciated then the destination would be another
>>> config_option (which you would then look up)
>>>
>>> if the type is 'ignore' then it would just be skipped over (possibly
>>> with
>>> a warning being logged that the config line is being ignored)
>>>
>>> this would also clean up some of the current cases where a boolean
>>> option
>>> doesn't honor on/off and true/false.
>>
>> True/false is NOT a valid boolean option. See
>>
>> http://git.adiscon.com/?p=rsyslog.git;a=blob;f=runtime/cfsysline.c;h=184c0d87
>> 47c5dcbbd8230782ca096e477738efa9;hb=HEAD#l308
>
> I was not remembering correctly then, maby it was yes/no vs on/off. I know
> I ran into something where what was documented didn't work and I changed
> it to another version of a boolean answer and it fixed the problem.
>
>>> For the second half of the config language (telling rsyslog what to do
>>> with the logs it has received) also has several variations in place and
>>> is
>>> showing issues.
>>>
>>> However I think that the right answer here is to end up implementing
>>> something like the parsing trees and then compile the other config
>>> language options into that tree to be consistant (and fast) under the
>>> covers, no matter which way the instructions are written (except when
>>> you
>>> have to use regex matches)
>>
>> I don't fully agree here with you. For example, the traditional PRI based
>> filter is lightning fast. I don't see any way it could be nearly as fast with
>> any unified approach. Right now, we have a "unified" filter structure, but it
>> contains three filter cases, each one adding potential power at the price of
>> decreased speed. I think we need to keep different types of filters in order
>> to have some lightning-fast ones. But more of this could be done under the
>> hood.
>
> My guess/expectation is that the tree-based parsing would be about the
> same speed as the traditional PRI based filter for choices made based on
> PRI, slowing down for other types of conditions only in that more of the
> message may need to be scanned (and if there is no selection at a level,
> skipping that level should be lightning fast). As a result, a set of
> filteres based soley on programname would be as fast as the current PRI
> filters.
>
> David Lang
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com
