After writing this, it just hit me that we could stay within the current config notation by introducing
$begin <type> $name <obj> $end <type> Constructs and don't change anything at the config language at all. The sample below could then be: ============================================================== $ModLoad ommysql # load the output driver (use ompgsql for PostgreSQL) $ModLoad imudp # network reception $UDPServerRun 514 # start a udp server at port 514 $ModLoad imuxsock # local message reception $WorkDirectory /rsyslog/work # default location for work (spool) files $Begin action $ActionQueueType LinkedList # use asynchronous processing $ActionQueueFileName dbq # set file name, also enables disk mode $ActionResumeRetryCount -1 # infinite retries on insert failure # for PostgreSQL replace :ommysql: by :ompgsql: below: *.* :ommysql:hostname,dbname,userid,password; $End action ============================================================== This would probably be possible to implement within the constraints of the current config parser. We could also add a directive $StrictScoping on to instruct rsyslog to disallow and $Action<whatever> directives outside of scoped blocks (this could be good to guard against accidential global directives). That still requires reworking of the internals, but would not need a new (real) grammar, so it would be considerable less work. I still need to see how it would work with more complex configs, but assuming we have things like $Begin Ruleset $Begin Rule $Begin Input I think the same paradigm could probably be used for everything (and that than would eliminate the need for a new grammar). Rainer > -----Original Message----- > From: [email protected] [mailto:rsyslog- > [email protected]] On Behalf Of Rainer Gerhards > Sent: Thursday, July 15, 2010 9:42 AM > To: rsyslog-users > Subject: Re: [rsyslog] NEW rsyslog.conf format > > Hi David, > > thanks for the well-crafted mail. My concerns for this proposal were > (and > are) mainly based on the "under the cover" changes. Other than that, I > think > you are right, except that it boils down to personal taste. > > Let me ignore the internal changes for now. > > Throwing in "{}" creates a very compact, ultra-brief config language. > But it > would definitely work. I just have to admit it does not fit my > *personal* > taste because it carries a lot of implicit scoping as well (like what > exactly > is meant to be scoped by the {} -- an action, or an input, or a > ruleset...). > But I think the semantics of this format are close to any other config > format > that fits rsyslog. So I think it is probably worth giving it a try, so > that > we make at least some progress. The only thing I really would like to > change > is to use "()" instead of "{}" because I would like to reserve "{}" for > future use, where it may fit the user's expectation better than simple > parenthesis. But I guess that's not really a problem. > > One thing though, that is on my feature list is that I would like to > use a > more standard parser, that means one that can becreated with lex and > Bison. > While the hand-crafted parser is fine, it always is more work to extend > and > enhance it. As the parser is no critical component, I'd prefer to use > the > simpler approach. However, I need to check if I can find a suitable > grammar > for this proposal. This also works on the assumption. > > I'd also just concentrate on actions for now. So to double-check a > fairly > simple config in that format would look as follows: > > ============================================================== > $ModLoad ommysql # load the output driver (use ompgsql for PostgreSQL) > $ModLoad imudp # network reception > $UDPServerRun 514 # start a udp server at port 514 > $ModLoad imuxsock # local message reception > > $WorkDirectory /rsyslog/work # default location for work (spool) files > > ( > $ActionQueueType LinkedList # use asynchronous processing > $ActionQueueFileName dbq # set file name, also enables disk mode > $ActionResumeRetryCount -1 # infinite retries on insert failure > # for PostgreSQL replace :ommysql: by :ompgsql: below: > *.* :ommysql:hostname,dbname,userid,password; > ) > ============================================================== > > This is based on the second example in > http://www.rsyslog.com/doc-rsyslog_high_database_rate.html > > I will probably also begin to look at the internal changes. As it > looks, we > need to make them in any case. So it doesn't hurt to start with them, > even > though there initially will be no external (user) visibility that they > are > made. > > But at first, I'll start looking at how this may be processed by a > standard > flex/bison parser. From the work I already did, I know this is not > easy, but > could probably work. > > Feedback appreciated. > > Rainer > > > -----Original Message----- > > From: [email protected] [mailto:rsyslog- > > [email protected]] On Behalf Of [email protected] > > Sent: Tuesday, July 13, 2010 7:30 PM > > To: rsyslog-users > > Subject: Re: [rsyslog] NEW rsyslog.conf format > > > > On Tue, 13 Jul 2010, Rainer Gerhards wrote: > > > > > To add some of the less obvious things: support for multiple > > listeners going > > > to different outputs, in an easy to use way. Support for > explicitely > > > specified concurrency (or serialization) for high-speed > environments, > > support > > > for different queues, and tying of different queues to different > > object > > > classes (inputs, message processors, actions). Flexibility to > support > > > configuration for even unexpected plugins that we may not even know > > about > > > (because some third party writes them and never publishes them). > > > > > > But I begin to agree that we, the community, as a whole have very > > different > > > needs. I have gotten the impression that it is probably a good idea > > to stop > > > the current effort and re-start it with a requirements definition. > I > > have > > > tried to digest the discussions on config format we had over the > > year, but > > > sometimes it looks like the only consensus we have is that the > > current format > > > is ugly and hard to use. The solutions are very wide-ranging. I > have > > even > > > briefly looked at syslog-ng, and seen a lot of the pain again that > > makes me > > > dislike that format (but I'll probably still have a closer look and > > will try > > > to write up my detailed position why I do not find buying into this > > format is > > > a good idea). > > > > > > All in all, I think the best way to re-start our thinking about the > > config > > > format is to set up a web site where we gather user feedback on > > > > > > a) what problems they have with the config format > > > (not what they just dislike, but real problems, an example > > > From myself: it is nearly impossible to get the sequence right > > > To bind inputs to the right rulesets AND use ruleset inclusion) > > > b) which alternatives they see > > > > > > With all this being on a web site, it may be possible to craft a > > better > > > decision in 6 to 9 month, assuming we were able to gain sufficient > > feedback > > > during that time. > > > > > > An alternative would be to create a config parser interface, so > that > > > everybody could code up his favorite config language. However, this > > seems to > > > be impractical, as many of the ideas floating around (Lua, syslog- > ng > > style) > > > require not just different config parsers, but a different rsyslog > > processing > > > core as well. Obviously a complete rewrite in the case of Lua, less > > for > > > syslog-ng, but still considerate. For the syslog-ng style we would > > need to > > > create named filters, something that I really find questionable. > > Also, we > > > would need to add an interim layer between inputs and rulesets, > > something > > > that eats performance. In any case, this are not config-parser only > > changes. > > > > > > Of course, I could just pick one of the alternatives. However, it > > doesn't > > > make sense to invest a couple of month to do the config system > > "right", if we > > > know that a lot of folks will still be unhappy after we have done > > this. > > > > one thing that's really good about the current rsyslog config is that > > it > > makes doing simple things simple, especially for people used to > classic > > syslog. > > > > as you say, we need to say what the problems there are with the > > existing > > config format and look at how to solve those. We don't necessarily > need > > to > > change everything. > > > > weaknesses that I know of > > > > inability to easily/clearly define if-then-else > > > > inability to easily/clearly define/use rulesets > > > > inability to easily have multiple conditions that go to the same > > destination (this can be implemented via rulesets, see above) > > > > unclear scope rules for config options > > > > inability to easily/clearly define per-input rules/filters (this is > > part > > of the scope problem above) > > > > > > I don't think that this requires an entirely new config language. I > > think > > that almost everything can be solved with a couple simple changes to > > the > > config language (although as we found when I proposed this on June 25 > > there are more drastic changes under the covers to > > check/correct/document > > what gets changed when a config option is processed) > > > > > > Ignoring for the moment the problem of changing how the config > options > > are > > processed (and the fact that you would need to know what data > > structures > > are managed/created/modified) what does the following proposal _not_ > > do? > > > > (pasted in from the mail on June 25 since that's now quite a ways > back > > in > > the archives ;-) > > > > > > > > I would propose the following (more or less in order of difficulty) > > > > introduce scoping > > > > whenever you see a "{" in the config file, save the current config > > options to a stack. when you see a "}" pop the config options from > the > > stack (undoing any changes in between) introduce statement blocks > > change the parser so that wherever it currently allows one action > > make > > it accept a {} block of actions (treat them as if they were multiple > > actions with & as the selector) > > > > introduce named actions > > > > something like sub NAME <action> to define and use NAME to use > > > > introduce if-then-else > > > > internally you could convert it to > > > > ruleset { > > if condition { > > block > > discard} > > { block } } > > > > introduct the ability to implcitly define a ruleset > > > > if an action is a condition (i.e. nested configutations) then > > implicitly > > create a new ruleset to resolve the nesting. > > > > > > with these capabilities available I think that this will allow for > > straightforward config files representing very complex configurations > > with > > very little change internally to rsyslog. > > > > I suspect that the result is going to have some bottlenecks in > complex > > configurations due to all the different rulesets and the passing of > > messages between them, but once the capability is in place in the > > config > > file the implementation under the covers could change later to be > > better > > optimized. > > > > as far as the rsyslog config being hard to understand, I think there > > are > > two conditions here. > > > > 1. very simple configs where the fact that rsyslog can use a > > traditional > > syslog config file (with a header on it that seldom needs to change) > is > > a > > huge advantage > > > > 2. complex configs where the inability to define scope and nest > things > > is > > a major problem and makes the resulting config hard to write. > > > > David Lang > > _______________________________________________ > > rsyslog mailing list > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > http://www.rsyslog.com > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com
