> -----Original Message----- > From: [email protected] [mailto:rsyslog- > [email protected]] On Behalf Of [email protected] > Sent: Friday, June 25, 2010 9:50 PM > To: rsyslog-users > Subject: Re: [rsyslog] NEW rsyslog.conf format > > On Fri, 25 Jun 2010, Rainer Gerhards wrote: > > > Hi David, > > > > I think your recommendations make a lot of sense. However, there is > one thing > > I need to clarify. You expect that all of this requires little work. > > Unfortunately, this is not the case. > > > > The current system has evolved from a very, very old system. So there > is > > nothing that I would really call a "parser" inside the system. The > config > > handler pulls one line at a time, and then looks at some keyword, and > stuffs > > the name-value pair to some registered handler. Then, the handler > does > > *something* (and it is fully up to the handler, and unknown to the > config > > processor, what this "something" may be). In many cases, it just sets > a > > global variable, in the more important cases it actually creates and > starts > > an action or creates and (potentially, but not necessarily) starts an > input. > > So there are no clear boundaries between what is reading the config > and what > > is starting up the system. All of this is mixed up in one large piece > (mess? > > ;)) of code. > > > > Introducing something like scoping is really, really problematic with > that > > system. For example, that requires that all plugins be changed, > because that > > violates the plugin interface. The main config processor does not > even know > > what the plugin may have done to a config statement, so it has no way > of > > keeping state of that. > > do you have a way of knowing what variables exist?
Often, but not always. Some handlers transform the input and store it in some other data structure. This is explicitely permitted. The config handler knows the directive, and if it modifies just some data element (and if so, which) or has such a "custom handler", in which cause it has no clue of what that does. The system was never designed for scoping, and thus does not provide any of the necessary facilities. > if you even have > that > much it would seem like you should be able to store everything and then > restore it at a later point. that way when the next handler runs it > will > see the 'earlier' version of a config. > > or are you saying that some of the data that would need to be scoped is > managed by the handlers in such a way that nothing else knows that that > data exists? Yup :) And as I said: a handler may do mach more. For example the InputTcpServerListe (or so) directive does not store the port to listen to but rather starts up the listener -- right at *that point* in config processing. > > In any case, creating a config parser that cleanly handles the existing > config language would be a good thing, and since you are not going to > eliminate the existing config language, it's probably a neccessary > thing > for long-term sanity in any case. Yes, definitely. And it is a prequisite for a) cleanly dropping privileges (Ubuntu drops privileges today against my recommendation and runs into one trouble after another because this is -- documentedly -- not properly supported yet) b) config reload on HUP (long way to go, but a two-step config read/apply is a necessary prequiste) plus maybe some others ;) Rainer > > David Lang > > > Based on this discussion, I think the proper route is to begin to > structure > > config processing and config activation. So that there exist a > parser, one > > that creates an in-memory representation of the config, but nothing > else. > > Then, in a second step, that in-memory representation is applied and > the > > system started up. > > > > This is a non-trivial piece of work. Among others, it requires > considerable > > changes to the plugin interface and a rewrite of all plugins. Also, > large > > parts of the current config system needs to be replaced. I expect > this takes > > until end of summer to complete (but I may be too pessimistic). As a > > (re-iterated) side-note, this is the reason I was always so hesitant > to > > change the config language. > > > > Once this is done, we will end up with exactly where we are now -- > form a > > pure users's PoV (no new capabilities, no faster or more reliable > code). > > However, we than have a basis where it is relatively simple to > introduce new > > config languages or extend the current language. > > > > My original thought was that I could do the redesign together with a > new > > config language, so that we would see some immediate benefit. > However, I now > > begin to think this is the wrong approach. I should build the basis > first, > > then it probably is very easy to define config parsers, which then > only need > > to create a proper in-memory representation of the system. I fear, > however, > > that I will probably need to change the plugin interface for each new > > language, so we should not do this too often ;) > > > > Rainer > > > >> -----Original Message----- > >> From: [email protected] [mailto:rsyslog- > >> [email protected]] On Behalf Of [email protected] > >> Sent: Friday, June 25, 2010 7:26 PM > >> To: rsyslog-users > >> Subject: Re: [rsyslog] RainerScript revited? NEW rsyslog.conf format > >> > >> On Fri, 25 Jun 2010, Rainer Gerhards wrote: > >> > >>> Reply to bath David's and Aaron's post: > >>> > >>>> -----Original Message----- > >>>> From: [email protected] [mailto:rsyslog- > >>>> [email protected]] On Behalf Of Aaron Wiebe > >>>> Sent: Friday, June 25, 2010 5:36 PM > >>>> To: rsyslog-users > >>>> Subject: Re: [rsyslog] RainerScript revited? NEW rsyslog.conf > format > >>>> > >>>> I agree, we're starting to get to the point of wondering why > again. > >>>> So here is my reason why: > >>>> > >>>> My basic concern is scoping. A lot of the current format is > >>>> order-specific, implicitly scoped configuration that makes it > >>>> extremely hard to work with. With more explicit scoping, I'm > happy. > >>> > >>> ... that's one reason for the config language. > >>> > >>> Next I see a large amount of questions that boil down to the > missing > >>> if/then/else functionality (as elaborated in another post). > >>> > >>> The functionality to use a single action in more than one rule is > >> missing > >>> (but can be emulated using omruleset). > >>> > >>> Config-reload on HUP with the current system is impossible (not a > >> problem of > >>> the config language, but of the engine). Same for full support for > >> dropping > >>> privileges. > >>> > >>> Probably a handful of other, minor things (one being the somewhat > >> annoying, > >>> always-heard complaint that the config language is hard to work > with, > >> ugly, > >>> unintuitive, ...). > >> > >> I would propose the following (more or less in order of difficulty) > >> > >> introduce scoping > >> > >> whenever you see a "{" in the config file, save the current > config > >> options to a stack. when you see a "}" pop the config options from > the > >> stack (undoing any changes in between) > >> > >> introduce statement blocks > >> > >> change the parser so that wherever it currently allows one action > >> make > >> it accept a {} block of actions (treat them as if they were multiple > >> actions with & as the selector) > >> > >> introduce named actions > >> > >> something like sub NAME <action> to define and use NAME to use > >> > >> introduce if-then-else > >> > >> internally you could convert it to > >> > >> ruleset { > >> if condition { > >> block > >> discard} > >> { block } > >> } > >> > >> introduct the ability to implcitly define a ruleset > >> > >> if an action is a condition (i.e. nested configutations) then > >> implicitly > >> create a new ruleset to resolve the nesting. > >> > >> > >> with these capabilities available I think that this will allow for > >> straightforward config files representing very complex > configurations > >> with > >> very little change internally to rsyslog. > >> > >> I suspect that the result is going to have some bottlenecks in > complex > >> configurations due to all the different rulesets and the passing of > >> messages between them, but once the capability is in place in the > >> config > >> file the implementation under the covers could change later to be > >> better > >> optimized. > >> > >> as far as the rsyslog config being hard to understand, I think there > >> are > >> two conditions here. > >> > >> 1. very simple configs where the fact that rsyslog can use a > >> traditional > >> syslog config file (with a header on it that seldom needs to change) > is > >> a > >> huge advantage > >> > >> 2. complex configs where the inability to define scope and nest > things > >> is > >> a major problem and makes the resulting config hard to write. > >> > >> David Lang > >> > >>> But, after all, I managed to use the same config language for years > >> now. When > >>> Adiscon considered changing its overall config language (probably > >> going XML > >>> now, but not yet decided), I thought this was a nice chance to get > >> some > >>> momentum for a new rsyslog config system. > >>> > >>> However, if the practical requirement boils down to scope (and it > >> looks that > >>> way), I am more than happy. All I need to do in that case is change > >> all > >>> config values to auto-reset after change. That means more config > code > >> must be > >>> written (copied), but you always know what is active. Probably a > good > >> week of > >>> work to do (plus a switch to enable the old behavior for > >> compatibility). I > >>> have no problem with that and could continue to look into some > "more > >>> interesting" things (from my POV, like performance). > >>>> > >>>> > >>>> And I know this may not be a popular idea, but really, I think it > >>>> should be considered -- syslog-ng's format is actually quite > >>>> straightforward and easily understood. Having a format similar > may > >> be > >>>> a problem for "product differentiation", but honestly, the syslog- > ng > >>>> method of inputs, outputs, and functions to tie those together is > >>>> quite appropriate for a logging application. I'm not saying we > >> should > >>>> use their exact format, but I do like lifting some of their better > >>>> config ideas. (I'm also partial to the bind-style/syslog-ng-style > >>>> config format.) > >>> > >>> The config language closely reflects the internal design. If I use > >> syslog-ng > >>> config, I tie myself into their concepts and restrict me to them. > >> This is not > >>> something I want to do. > >>> > >>> Rainer > >>>> > >>>> -Aaron > >>>> > >>>> On Fri, Jun 25, 2010 at 11:29 AM, <[email protected]> wrote: > >>>>> On Fri, 25 Jun 2010, Rainer Gerhards wrote: > >>>>> > >>>>>> After all formats seem to have at least some problems attached, > I > >>>> decided to > >>>>>> have a look at my original RainerScript idea. Note that the > >> current > >>>> (partial) > >>>>>> implementation is NOT suitable for very high performance. > However, > >>>> that is > >>>>>> something that can be changed. > >>>>>> > >>>>>> To understand why a language, and why a specialized one, we need > >> to > >>>> look at > >>>>>> rsyslogd from a different angle: to me, rsyslogd is high > >> performance > >>>>>> processor good at shuffling messages from an input to an output > >> and > >>>> applying > >>>>>> transformations while doing so. There is conditional logic > >> involved > >>>> to craft > >>>>>> the path a specific message takes. > >>>>>> > >>>>>> So one can think of rsyslog like a specialized computer whom's > >>>> instruction > >>>>>> set is optimized for that purpose (I guess that is what David > was > >>>> referring > >>>>>> to when he talked about "functions"). > >>>>>> > >>>>>> With that view, the configuration file is actually a programming > >>>> language: > >>>>>> one that specifies how to process message. Reading the config > file > >>>> can then > >>>>>> be compared to a typical compilation, where the generated object > >>>> code is > >>>>>> actually targeted towards "rsyslog machine instructions". > >>>>>> > >>>>>> Obviously, to program that very special machine, we do need a > >>>> specialized > >>>>>> programming language as well. Alternatively, one may think of > this > >>>> language > >>>>>> as being the "assembly language" of rsyslog, which closely > matches > >>>> the > >>>>>> instruction set and thus can be used to write very efficient > >>>> programs (but > >>>>>> requires some skills for non-trivial cases). > >>>>>> > >>>>>> Having said this, all language constructs must translate > natively > >> to > >>>> rsyslog > >>>>>> engine objects. At the same time, they should be very simple to > >> use > >>>> for > >>>>>> novice users while providing all the expressiveness an expert > user > >>>> needs. > >>>>>> This most importantly means that the config language must be > easy > >> to > >>>> read and > >>>>>> edit by a human. > >>>>>> [Michael: *this* actually are the requirements for the config > >>>> language, > >>>>>> together with the other long post this morning -- I am not sure > if > >> I > >>>> find > >>>>>> time to consolidate these two today.] > >>>>>> > >>>>>> Now please have a look at a new sample config: > >>>>>> > >>>>>> http://download.rsyslog.com/rainerscript_rsyslog.conf > >>>>>> > >>>>>> This time, I have not only included a hypothetical complex > >>>> configuration, but > >>>>>> also a very simple standard config - just so that we can see how > >>>> verbose it > >>>>>> gets. All configs are inside a single file -- you need to scroll > >>>> down. > >>>>>> > >>>>>> All in all, I have to admit I begin to like the RainerScript > >>>> approach once > >>>>>> again. Unfortunately, it obviously is the solution that requires > >>>> more work > >>>>>> than any others (because I need to craft the grammar and the > >> parser > >>>> myself). > >>>>> > >>>>> after reading this I am wondering why everything needs to change. > >>>>> > >>>>> what concepts are missing in the current config language that you > >> are > >>>>> needing to add > >>>>> > >>>>> as I see it you need to add > >>>>> > >>>>> ability to define blocks of actions > >>>>> currently this is done by changing the ruleset, this needs to > be > >>>> made > >>>>> more explicit and obvious > >>>>> > >>>>> a block of actions may be a single action that's used more than > >>>> once > >>>>> > >>>>> > >>>>> ability to define if-then-else not just if-then > >>>>> > >>>>> combining these two to make the if-then or if-then-else do blocks > >> of > >>>>> actions. > >>>>> > >>>>> what else is missing? > >>>>> > >>>>> if you were going to a standard config language that other things > >>>> could > >>>>> understand then changing everything to fit that language makes > >> sense, > >>>> but > >>>>> if it's going to be another custom config parser what's the > benifit > >>>> of > >>>>> starting from scratch rather than just adding the features to the > >>>> existing > >>>>> one (since you will have to maintain the existing one anyway) > >>>>> > >>>>> David Lang > >>>>> > >>>>> P.S. is the rest of adiscon going to just go with XML? > >>>>> > >>>>> > >>>>> _______________________________________________ > >>>>> rsyslog mailing list > >>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog > >>>>> http://www.rsyslog.com > >>>>> > >>>> _______________________________________________ > >>>> rsyslog mailing list > >>>> http://lists.adiscon.net/mailman/listinfo/rsyslog > >>>> http://www.rsyslog.com > >>> _______________________________________________ > >>> rsyslog mailing list > >>> http://lists.adiscon.net/mailman/listinfo/rsyslog > >>> http://www.rsyslog.com > >>> > > _______________________________________________ > > rsyslog mailing list > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > http://www.rsyslog.com > > _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com
