Hi David, I think your recommendations make a lot of sense. However, there is one thing I need to clarify. You expect that all of this requires little work. Unfortunately, this is not the case.
The current system has evolved from a very, very old system. So there is nothing that I would really call a "parser" inside the system. The config handler pulls one line at a time, and then looks at some keyword, and stuffs the name-value pair to some registered handler. Then, the handler does *something* (and it is fully up to the handler, and unknown to the config processor, what this "something" may be). In many cases, it just sets a global variable, in the more important cases it actually creates and starts an action or creates and (potentially, but not necessarily) starts an input. So there are no clear boundaries between what is reading the config and what is starting up the system. All of this is mixed up in one large piece (mess? ;)) of code. Introducing something like scoping is really, really problematic with that system. For example, that requires that all plugins be changed, because that violates the plugin interface. The main config processor does not even know what the plugin may have done to a config statement, so it has no way of keeping state of that. Based on this discussion, I think the proper route is to begin to structure config processing and config activation. So that there exist a parser, one that creates an in-memory representation of the config, but nothing else. Then, in a second step, that in-memory representation is applied and the system started up. This is a non-trivial piece of work. Among others, it requires considerable changes to the plugin interface and a rewrite of all plugins. Also, large parts of the current config system needs to be replaced. I expect this takes until end of summer to complete (but I may be too pessimistic). As a (re-iterated) side-note, this is the reason I was always so hesitant to change the config language. Once this is done, we will end up with exactly where we are now -- form a pure users's PoV (no new capabilities, no faster or more reliable code). However, we than have a basis where it is relatively simple to introduce new config languages or extend the current language. My original thought was that I could do the redesign together with a new config language, so that we would see some immediate benefit. However, I now begin to think this is the wrong approach. I should build the basis first, then it probably is very easy to define config parsers, which then only need to create a proper in-memory representation of the system. I fear, however, that I will probably need to change the plugin interface for each new language, so we should not do this too often ;) Rainer > -----Original Message----- > From: [email protected] [mailto:rsyslog- > [email protected]] On Behalf Of [email protected] > Sent: Friday, June 25, 2010 7:26 PM > To: rsyslog-users > Subject: Re: [rsyslog] RainerScript revited? NEW rsyslog.conf format > > On Fri, 25 Jun 2010, Rainer Gerhards wrote: > > > Reply to bath David's and Aaron's post: > > > >> -----Original Message----- > >> From: [email protected] [mailto:rsyslog- > >> [email protected]] On Behalf Of Aaron Wiebe > >> Sent: Friday, June 25, 2010 5:36 PM > >> To: rsyslog-users > >> Subject: Re: [rsyslog] RainerScript revited? NEW rsyslog.conf format > >> > >> I agree, we're starting to get to the point of wondering why again. > >> So here is my reason why: > >> > >> My basic concern is scoping. A lot of the current format is > >> order-specific, implicitly scoped configuration that makes it > >> extremely hard to work with. With more explicit scoping, I'm happy. > > > > ... that's one reason for the config language. > > > > Next I see a large amount of questions that boil down to the missing > > if/then/else functionality (as elaborated in another post). > > > > The functionality to use a single action in more than one rule is > missing > > (but can be emulated using omruleset). > > > > Config-reload on HUP with the current system is impossible (not a > problem of > > the config language, but of the engine). Same for full support for > dropping > > privileges. > > > > Probably a handful of other, minor things (one being the somewhat > annoying, > > always-heard complaint that the config language is hard to work with, > ugly, > > unintuitive, ...). > > I would propose the following (more or less in order of difficulty) > > introduce scoping > > whenever you see a "{" in the config file, save the current config > options to a stack. when you see a "}" pop the config options from the > stack (undoing any changes in between) > > introduce statement blocks > > change the parser so that wherever it currently allows one action > make > it accept a {} block of actions (treat them as if they were multiple > actions with & as the selector) > > introduce named actions > > something like sub NAME <action> to define and use NAME to use > > introduce if-then-else > > internally you could convert it to > > ruleset { > if condition { > block > discard} > { block } > } > > introduct the ability to implcitly define a ruleset > > if an action is a condition (i.e. nested configutations) then > implicitly > create a new ruleset to resolve the nesting. > > > with these capabilities available I think that this will allow for > straightforward config files representing very complex configurations > with > very little change internally to rsyslog. > > I suspect that the result is going to have some bottlenecks in complex > configurations due to all the different rulesets and the passing of > messages between them, but once the capability is in place in the > config > file the implementation under the covers could change later to be > better > optimized. > > as far as the rsyslog config being hard to understand, I think there > are > two conditions here. > > 1. very simple configs where the fact that rsyslog can use a > traditional > syslog config file (with a header on it that seldom needs to change) is > a > huge advantage > > 2. complex configs where the inability to define scope and nest things > is > a major problem and makes the resulting config hard to write. > > David Lang > > > But, after all, I managed to use the same config language for years > now. When > > Adiscon considered changing its overall config language (probably > going XML > > now, but not yet decided), I thought this was a nice chance to get > some > > momentum for a new rsyslog config system. > > > > However, if the practical requirement boils down to scope (and it > looks that > > way), I am more than happy. All I need to do in that case is change > all > > config values to auto-reset after change. That means more config code > must be > > written (copied), but you always know what is active. Probably a good > week of > > work to do (plus a switch to enable the old behavior for > compatibility). I > > have no problem with that and could continue to look into some "more > > interesting" things (from my POV, like performance). > >> > >> > >> And I know this may not be a popular idea, but really, I think it > >> should be considered -- syslog-ng's format is actually quite > >> straightforward and easily understood. Having a format similar may > be > >> a problem for "product differentiation", but honestly, the syslog-ng > >> method of inputs, outputs, and functions to tie those together is > >> quite appropriate for a logging application. I'm not saying we > should > >> use their exact format, but I do like lifting some of their better > >> config ideas. (I'm also partial to the bind-style/syslog-ng-style > >> config format.) > > > > The config language closely reflects the internal design. If I use > syslog-ng > > config, I tie myself into their concepts and restrict me to them. > This is not > > something I want to do. > > > > Rainer > >> > >> -Aaron > >> > >> On Fri, Jun 25, 2010 at 11:29 AM, <[email protected]> wrote: > >>> On Fri, 25 Jun 2010, Rainer Gerhards wrote: > >>> > >>>> After all formats seem to have at least some problems attached, I > >> decided to > >>>> have a look at my original RainerScript idea. Note that the > current > >> (partial) > >>>> implementation is NOT suitable for very high performance. However, > >> that is > >>>> something that can be changed. > >>>> > >>>> To understand why a language, and why a specialized one, we need > to > >> look at > >>>> rsyslogd from a different angle: to me, rsyslogd is high > performance > >>>> processor good at shuffling messages from an input to an output > and > >> applying > >>>> transformations while doing so. There is conditional logic > involved > >> to craft > >>>> the path a specific message takes. > >>>> > >>>> So one can think of rsyslog like a specialized computer whom's > >> instruction > >>>> set is optimized for that purpose (I guess that is what David was > >> referring > >>>> to when he talked about "functions"). > >>>> > >>>> With that view, the configuration file is actually a programming > >> language: > >>>> one that specifies how to process message. Reading the config file > >> can then > >>>> be compared to a typical compilation, where the generated object > >> code is > >>>> actually targeted towards "rsyslog machine instructions". > >>>> > >>>> Obviously, to program that very special machine, we do need a > >> specialized > >>>> programming language as well. Alternatively, one may think of this > >> language > >>>> as being the "assembly language" of rsyslog, which closely matches > >> the > >>>> instruction set and thus can be used to write very efficient > >> programs (but > >>>> requires some skills for non-trivial cases). > >>>> > >>>> Having said this, all language constructs must translate natively > to > >> rsyslog > >>>> engine objects. At the same time, they should be very simple to > use > >> for > >>>> novice users while providing all the expressiveness an expert user > >> needs. > >>>> This most importantly means that the config language must be easy > to > >> read and > >>>> edit by a human. > >>>> [Michael: *this* actually are the requirements for the config > >> language, > >>>> together with the other long post this morning -- I am not sure if > I > >> find > >>>> time to consolidate these two today.] > >>>> > >>>> Now please have a look at a new sample config: > >>>> > >>>> http://download.rsyslog.com/rainerscript_rsyslog.conf > >>>> > >>>> This time, I have not only included a hypothetical complex > >> configuration, but > >>>> also a very simple standard config - just so that we can see how > >> verbose it > >>>> gets. All configs are inside a single file -- you need to scroll > >> down. > >>>> > >>>> All in all, I have to admit I begin to like the RainerScript > >> approach once > >>>> again. Unfortunately, it obviously is the solution that requires > >> more work > >>>> than any others (because I need to craft the grammar and the > parser > >> myself). > >>> > >>> after reading this I am wondering why everything needs to change. > >>> > >>> what concepts are missing in the current config language that you > are > >>> needing to add > >>> > >>> as I see it you need to add > >>> > >>> ability to define blocks of actions > >>> currently this is done by changing the ruleset, this needs to be > >> made > >>> more explicit and obvious > >>> > >>> a block of actions may be a single action that's used more than > >> once > >>> > >>> > >>> ability to define if-then-else not just if-then > >>> > >>> combining these two to make the if-then or if-then-else do blocks > of > >>> actions. > >>> > >>> what else is missing? > >>> > >>> if you were going to a standard config language that other things > >> could > >>> understand then changing everything to fit that language makes > sense, > >> but > >>> if it's going to be another custom config parser what's the benifit > >> of > >>> starting from scratch rather than just adding the features to the > >> existing > >>> one (since you will have to maintain the existing one anyway) > >>> > >>> David Lang > >>> > >>> P.S. is the rest of adiscon going to just go with XML? > >>> > >>> > >>> _______________________________________________ > >>> rsyslog mailing list > >>> http://lists.adiscon.net/mailman/listinfo/rsyslog > >>> http://www.rsyslog.com > >>> > >> _______________________________________________ > >> rsyslog mailing list > >> http://lists.adiscon.net/mailman/listinfo/rsyslog > >> http://www.rsyslog.com > > _______________________________________________ > > rsyslog mailing list > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > http://www.rsyslog.com > > _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com
