Hi David, yes, that makes much more sense to me. Let's call that filter expression tree for the time being (the parse tree used for normalization has some similar ideas, but a totally different scope and semantics).
While I now see where you are heading, I fail to see how this could sufficiently easy be implemented for a real (and complex) configuration. Let's look at this example conf: mail.* /Var/log/mail.log if programname startswith abc log to /abc if programname startswith 'acc' and MSG contains 'error' log to /acc # we don't want debug messages from here on *.debug ~ if programname startswith 'acc' or (programname contains 'bde' and (facility > 1 and facility < 5)) log to @1.1.1.1 if programname startswith bcd or MSG startswith '123' or MSG contains 'error2' log to /bcd *.* /var/log/catchall I simply cannot envision how you would store this as a tree that does not look like what we have today. Rainer > -----Original Message----- > From: [email protected] [mailto:rsyslog- > [email protected]] On Behalf Of [email protected] > Sent: Sunday, February 28, 2010 2:02 PM > To: rsyslog-users > Subject: Re: [rsyslog] Feedback requested: inconsistency in config > statements > > On Sun, 28 Feb 2010, Rainer Gerhards wrote: > > > Quickly from the mobile: pri based filters require *excatly* one > single word (!) lookup currently. You can't beat that. > > > > I am not totally sure how you think about parse trees. In my view, > parsin and filteing are two different things./stages. Maybe we have > different concepts on our minds? > > I am probably using the wrong terminology here. > > say you have a set of rules that say > > if programname startswith abc log to /abc > if programname startswith acc log to /acc > if programname startswith acc log to @1.1.1.1 > if programname startswith bcd log to /bcd > > and assuming that these were the only possible programnames for > simplicity > in the explination. > > the filtering logic would go somthing like this > > the rules would compile into a tree > > -a-b -> /abc > -c -> /acc,@1.1.1.1 > -b -> /bcd > > receive programname abc > I have no facility/Pri filter rules > I have no time filter rules > I have no system filter rules > I have programname filter rules > look at the first character of the programname > it's "a", I have more than one rule that fits that. > look at the second character of the programname > it's a "b", There are no other decisions to make, > invoke the action /abc > > receive progranmane bcd > I have no facility/PRI filter rules > I have no time filter rules > I have no system filter rules > I have programname filter rules > look at the first character of the programname > it's a "b", There are no other decisions to make, > invoke the action /bcd > > receive programname acc > I have no facility/Pri filter rules > I have no time filter rules > I have no system filter rules > I have programname filter rules > look at the first character of the programname > it's "a", I have more than one rule that fits that. > look at the second character of the programname > it's a "c", There are no other decisions to make, > invoke the action /acc and the action @1.1.1.1 > > similarly the facility/pri filtering rules would be either compiled > into a > tree, or (since it's 256 entries total) into a lookup table with > pointers > to the list of actions to take for that entry > > the idea is to spend the time at startup to create the tree that > represents the ruleset in order to speed up the processing of each > individual message. > > the real tree would be a bit more complicated than I describe here as > it > needs to have 'anything else' entries between the known entries (which > means that it would not be able to shortcut quite as much as I > describe), > and have provision for 'do a more complicated thing (like regex) here > and > if it matches continue'. but except for regex matches, this would make > processing the rules pretty much independant of how many rules there > were > or how complex the ruleset is. > > there doe snot need to be one single programname filter tree, if you > had a > rule that said > if pri = info and programname startswith def then def > > the pri tree/table would have an entry for info that would point to a > filter tree for programname that just had the check for def in it > > am I makeing any more sense now? > > David Lang > > > rainer > > > > ----- Urspr?ngliche Nachricht ----- > > Von: "[email protected]" <[email protected]> > > An: "rsyslog-users" <[email protected]> > > Gesendet: 28.02.10 11:28 > > Betreff: Re: [rsyslog] Feedback requested: inconsistency in config > statements > > > > On Sun, 28 Feb 2010, Rainer Gerhards wrote: > > > >>> > >>> if the type is depriciated then the destination would be another > >>> config_option (which you would then look up) > >>> > >>> if the type is 'ignore' then it would just be skipped over > (possibly > >>> with > >>> a warning being logged that the config line is being ignored) > >>> > >>> this would also clean up some of the current cases where a boolean > >>> option > >>> doesn't honor on/off and true/false. > >> > >> True/false is NOT a valid boolean option. See > >> > >> > http://git.adiscon.com/?p=rsyslog.git;a=blob;f=runtime/cfsysline.c;h=18 > 4c0d87 > >> 47c5dcbbd8230782ca096e477738efa9;hb=HEAD#l308 > > > > I was not remembering correctly then, maby it was yes/no vs on/off. I > know > > I ran into something where what was documented didn't work and I > changed > > it to another version of a boolean answer and it fixed the problem. > > > >>> For the second half of the config language (telling rsyslog what to > do > >>> with the logs it has received) also has several variations in place > and > >>> is > >>> showing issues. > >>> > >>> However I think that the right answer here is to end up > implementing > >>> something like the parsing trees and then compile the other config > >>> language options into that tree to be consistant (and fast) under > the > >>> covers, no matter which way the instructions are written (except > when > >>> you > >>> have to use regex matches) > >> > >> I don't fully agree here with you. For example, the traditional PRI > based > >> filter is lightning fast. I don't see any way it could be nearly as > fast with > >> any unified approach. Right now, we have a "unified" filter > structure, but it > >> contains three filter cases, each one adding potential power at the > price of > >> decreased speed. I think we need to keep different types of filters > in order > >> to have some lightning-fast ones. But more of this could be done > under the > >> hood. > > > > My guess/expectation is that the tree-based parsing would be about > the > > same speed as the traditional PRI based filter for choices made based > on > > PRI, slowing down for other types of conditions only in that more of > the > > message may need to be scanned (and if there is no selection at a > level, > > skipping that level should be lightning fast). As a result, a set of > > filteres based soley on programname would be as fast as the current > PRI > > filters. > > > > David Lang > > _______________________________________________ > > rsyslog mailing list > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > http://www.rsyslog.com > > _______________________________________________ > > rsyslog mailing list > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > http://www.rsyslog.com > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com
