On Thu, 15 Jul 2010, Rainer Gerhards wrote:
> Hi David,
>
> thanks for the well-crafted mail. My concerns for this proposal were (and
> are) mainly based on the "under the cover" changes. Other than that, I think
> you are right, except that it boils down to personal taste.
>
> Let me ignore the internal changes for now.
>
> Throwing in "{}" creates a very compact, ultra-brief config language. But it
> would definitely work. I just have to admit it does not fit my *personal*
> taste because it carries a lot of implicit scoping as well (like what exactly
> is meant to be scoped by the {} -- an action, or an input, or a ruleset...).
> But I think the semantics of this format are close to any other config format
> that fits rsyslog. So I think it is probably worth giving it a try, so that
> we make at least some progress. The only thing I really would like to change
> is to use "()" instead of "{}" because I would like to reserve "{}" for
> future use, where it may fit the user's expectation better than simple
> parenthesis. But I guess that's not really a problem.
sure, I suggested {} in part from habit as they are used to defien blocks
in several languages while () tends to be used for parameters to a
function, but this is a cosmetic change (and easy to tweak once a
proof-of-concept parser is available)
> One thing though, that is on my feature list is that I would like to use a
> more standard parser, that means one that can becreated with lex and Bison.
> While the hand-crafted parser is fine, it always is more work to extend and
> enhance it. As the parser is no critical component, I'd prefer to use the
> simpler approach. However, I need to check if I can find a suitable grammar
> for this proposal. This also works on the assumption.
I don't see this as a problem. It's work to define a parser, but I don't
think that the current grammer is significantly harder to put into a
lex/bison parser than any other (the exception being the implicit scoping
rules)
> I'd also just concentrate on actions for now. So to double-check a fairly
> simple config in that format would look as follows:
>
> ==============================================================
> $ModLoad ommysql # load the output driver (use ompgsql for PostgreSQL)
> $ModLoad imudp # network reception
> $UDPServerRun 514 # start a udp server at port 514
> $ModLoad imuxsock # local message reception
>
> $WorkDirectory /rsyslog/work # default location for work (spool) files
>
> (
> $ActionQueueType LinkedList # use asynchronous processing
> $ActionQueueFileName dbq # set file name, also enables disk mode
> $ActionResumeRetryCount -1 # infinite retries on insert failure
> # for PostgreSQL replace :ommysql: by :ompgsql: below:
> *.* :ommysql:hostname,dbname,userid,password;
> )
> ==============================================================
>
> This is based on the second example in
> http://www.rsyslog.com/doc-rsyslog_high_database_rate.html
looks sane to me.
one question, does the comment really mean what it sounds like on this
line?
$ActionQueueType LinkedList # use asynchronous processing
This makes it sound like you don't do async processing if you use the
default queue type. I didn't think that that was the case.
> I will probably also begin to look at the internal changes. As it looks, we
> need to make them in any case. So it doesn't hurt to start with them, even
> though there initially will be no external (user) visibility that they are
> made.
I agree, the changes that I think are going to be needed are as much
documentation as they are functional changes. Makeing sure that we know
what variables are changed by each config line that's possible, and making
it so that these changes are done in a way that they can be swapped out
for a different set of values as scope changes happen.
David Lang
> But at first, I'll start looking at how this may be processed by a standard
> flex/bison parser. From the work I already did, I know this is not easy, but
> could probably work.
>
> Feedback appreciated.
>
> Rainer
>
>> -----Original Message-----
>> From: [email protected] [mailto:rsyslog-
>> [email protected]] On Behalf Of [email protected]
>> Sent: Tuesday, July 13, 2010 7:30 PM
>> To: rsyslog-users
>> Subject: Re: [rsyslog] NEW rsyslog.conf format
>>
>> On Tue, 13 Jul 2010, Rainer Gerhards wrote:
>>
>>> To add some of the less obvious things: support for multiple
>> listeners going
>>> to different outputs, in an easy to use way. Support for explicitely
>>> specified concurrency (or serialization) for high-speed environments,
>> support
>>> for different queues, and tying of different queues to different
>> object
>>> classes (inputs, message processors, actions). Flexibility to support
>>> configuration for even unexpected plugins that we may not even know
>> about
>>> (because some third party writes them and never publishes them).
>>>
>>> But I begin to agree that we, the community, as a whole have very
>> different
>>> needs. I have gotten the impression that it is probably a good idea
>> to stop
>>> the current effort and re-start it with a requirements definition. I
>> have
>>> tried to digest the discussions on config format we had over the
>> year, but
>>> sometimes it looks like the only consensus we have is that the
>> current format
>>> is ugly and hard to use. The solutions are very wide-ranging. I have
>> even
>>> briefly looked at syslog-ng, and seen a lot of the pain again that
>> makes me
>>> dislike that format (but I'll probably still have a closer look and
>> will try
>>> to write up my detailed position why I do not find buying into this
>> format is
>>> a good idea).
>>>
>>> All in all, I think the best way to re-start our thinking about the
>> config
>>> format is to set up a web site where we gather user feedback on
>>>
>>> a) what problems they have with the config format
>>> (not what they just dislike, but real problems, an example
>>> From myself: it is nearly impossible to get the sequence right
>>> To bind inputs to the right rulesets AND use ruleset inclusion)
>>> b) which alternatives they see
>>>
>>> With all this being on a web site, it may be possible to craft a
>> better
>>> decision in 6 to 9 month, assuming we were able to gain sufficient
>> feedback
>>> during that time.
>>>
>>> An alternative would be to create a config parser interface, so that
>>> everybody could code up his favorite config language. However, this
>> seems to
>>> be impractical, as many of the ideas floating around (Lua, syslog-ng
>> style)
>>> require not just different config parsers, but a different rsyslog
>> processing
>>> core as well. Obviously a complete rewrite in the case of Lua, less
>> for
>>> syslog-ng, but still considerate. For the syslog-ng style we would
>> need to
>>> create named filters, something that I really find questionable.
>> Also, we
>>> would need to add an interim layer between inputs and rulesets,
>> something
>>> that eats performance. In any case, this are not config-parser only
>> changes.
>>>
>>> Of course, I could just pick one of the alternatives. However, it
>> doesn't
>>> make sense to invest a couple of month to do the config system
>> "right", if we
>>> know that a lot of folks will still be unhappy after we have done
>> this.
>>
>> one thing that's really good about the current rsyslog config is that
>> it
>> makes doing simple things simple, especially for people used to classic
>> syslog.
>>
>> as you say, we need to say what the problems there are with the
>> existing
>> config format and look at how to solve those. We don't necessarily need
>> to
>> change everything.
>>
>> weaknesses that I know of
>>
>> inability to easily/clearly define if-then-else
>>
>> inability to easily/clearly define/use rulesets
>>
>> inability to easily have multiple conditions that go to the same
>> destination (this can be implemented via rulesets, see above)
>>
>> unclear scope rules for config options
>>
>> inability to easily/clearly define per-input rules/filters (this is
>> part
>> of the scope problem above)
>>
>>
>> I don't think that this requires an entirely new config language. I
>> think
>> that almost everything can be solved with a couple simple changes to
>> the
>> config language (although as we found when I proposed this on June 25
>> there are more drastic changes under the covers to
>> check/correct/document
>> what gets changed when a config option is processed)
>>
>>
>> Ignoring for the moment the problem of changing how the config options
>> are
>> processed (and the fact that you would need to know what data
>> structures
>> are managed/created/modified) what does the following proposal _not_
>> do?
>>
>> (pasted in from the mail on June 25 since that's now quite a ways back
>> in
>> the archives ;-)
>>
>>
>>
>> I would propose the following (more or less in order of difficulty)
>>
>> introduce scoping
>>
>> whenever you see a "{" in the config file, save the current config
>> options to a stack. when you see a "}" pop the config options from the
>> stack (undoing any changes in between) introduce statement blocks
>> change the parser so that wherever it currently allows one action
>> make
>> it accept a {} block of actions (treat them as if they were multiple
>> actions with & as the selector)
>>
>> introduce named actions
>>
>> something like sub NAME <action> to define and use NAME to use
>>
>> introduce if-then-else
>>
>> internally you could convert it to
>>
>> ruleset {
>> if condition {
>> block
>> discard}
>> { block } }
>>
>> introduct the ability to implcitly define a ruleset
>>
>> if an action is a condition (i.e. nested configutations) then
>> implicitly
>> create a new ruleset to resolve the nesting.
>>
>>
>> with these capabilities available I think that this will allow for
>> straightforward config files representing very complex configurations
>> with
>> very little change internally to rsyslog.
>>
>> I suspect that the result is going to have some bottlenecks in complex
>> configurations due to all the different rulesets and the passing of
>> messages between them, but once the capability is in place in the
>> config
>> file the implementation under the covers could change later to be
>> better
>> optimized.
>>
>> as far as the rsyslog config being hard to understand, I think there
>> are
>> two conditions here.
>>
>> 1. very simple configs where the fact that rsyslog can use a
>> traditional
>> syslog config file (with a header on it that seldom needs to change) is
>> a
>> huge advantage
>>
>> 2. complex configs where the inability to define scope and nest things
>> is
>> a major problem and makes the resulting config hard to write.
>>
>> David Lang
>> _______________________________________________
>> rsyslog mailing list
>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>> http://www.rsyslog.com
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com
>
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com
