On Thu, 15 Jul 2010, Rainer Gerhards wrote:
> After writing this, it just hit me that we could stay within the current
> config notation by introducing
>
> $begin <type>
> $end <type>
I have no problem with this. I view is as symatically eqivalent to the
{}(), just more characters to type ;-)
> $name <obj>
I had even suggested NAME <name> block and USE <name> syntaxes
the only question is if we need to explicitly state the type or if it's
good enough to be able to just nest the scope. I think you can get away
with just nesting the scope, but I could be wrong and if so defining the
type is a fairly cheap way of working around the issue.
> The sample below could then be:
>
> ==============================================================
> $ModLoad ommysql # load the output driver (use ompgsql for PostgreSQL)
> $ModLoad imudp # network reception
> $UDPServerRun 514 # start a udp server at port 514
> $ModLoad imuxsock # local message reception
>
> $WorkDirectory /rsyslog/work # default location for work (spool) files
>
> $Begin action
> $ActionQueueType LinkedList # use asynchronous processing
> $ActionQueueFileName dbq # set file name, also enables disk mode
> $ActionResumeRetryCount -1 # infinite retries on insert failure
> # for PostgreSQL replace :ommysql: by :ompgsql: below:
> *.* :ommysql:hostname,dbname,userid,password;
> $End action
> ==============================================================
>
> This would probably be possible to implement within the constraints of the
> current config parser. We could also add a directive
>
> $StrictScoping on
>
> to instruct rsyslog to disallow and $Action<whatever> directives outside of
> scoped blocks (this could be good to guard against accidential global
> directives).
probably a good idea.
> That still requires reworking of the internals, but would not need a new
> (real) grammar, so it would be considerable less work. I still need to see
> how it would work with more complex configs, but assuming we have things like
>
> $Begin Ruleset
> $Begin Rule
> $Begin Input
>
> I think the same paradigm could probably be used for everything (and that
> than would eliminate the need for a new grammar).
sounds good.
David Lang
> Rainer
>
>> -----Original Message-----
>> From: [email protected] [mailto:rsyslog-
>> [email protected]] On Behalf Of Rainer Gerhards
>> Sent: Thursday, July 15, 2010 9:42 AM
>> To: rsyslog-users
>> Subject: Re: [rsyslog] NEW rsyslog.conf format
>>
>> Hi David,
>>
>> thanks for the well-crafted mail. My concerns for this proposal were
>> (and
>> are) mainly based on the "under the cover" changes. Other than that, I
>> think
>> you are right, except that it boils down to personal taste.
>>
>> Let me ignore the internal changes for now.
>>
>> Throwing in "{}" creates a very compact, ultra-brief config language.
>> But it
>> would definitely work. I just have to admit it does not fit my
>> *personal*
>> taste because it carries a lot of implicit scoping as well (like what
>> exactly
>> is meant to be scoped by the {} -- an action, or an input, or a
>> ruleset...).
>> But I think the semantics of this format are close to any other config
>> format
>> that fits rsyslog. So I think it is probably worth giving it a try, so
>> that
>> we make at least some progress. The only thing I really would like to
>> change
>> is to use "()" instead of "{}" because I would like to reserve "{}" for
>> future use, where it may fit the user's expectation better than simple
>> parenthesis. But I guess that's not really a problem.
>>
>> One thing though, that is on my feature list is that I would like to
>> use a
>> more standard parser, that means one that can becreated with lex and
>> Bison.
>> While the hand-crafted parser is fine, it always is more work to extend
>> and
>> enhance it. As the parser is no critical component, I'd prefer to use
>> the
>> simpler approach. However, I need to check if I can find a suitable
>> grammar
>> for this proposal. This also works on the assumption.
>>
>> I'd also just concentrate on actions for now. So to double-check a
>> fairly
>> simple config in that format would look as follows:
>>
>> ==============================================================
>> $ModLoad ommysql # load the output driver (use ompgsql for PostgreSQL)
>> $ModLoad imudp # network reception
>> $UDPServerRun 514 # start a udp server at port 514
>> $ModLoad imuxsock # local message reception
>>
>> $WorkDirectory /rsyslog/work # default location for work (spool) files
>>
>> (
>> $ActionQueueType LinkedList # use asynchronous processing
>> $ActionQueueFileName dbq # set file name, also enables disk mode
>> $ActionResumeRetryCount -1 # infinite retries on insert failure
>> # for PostgreSQL replace :ommysql: by :ompgsql: below:
>> *.* :ommysql:hostname,dbname,userid,password;
>> )
>> ==============================================================
>>
>> This is based on the second example in
>> http://www.rsyslog.com/doc-rsyslog_high_database_rate.html
>>
>> I will probably also begin to look at the internal changes. As it
>> looks, we
>> need to make them in any case. So it doesn't hurt to start with them,
>> even
>> though there initially will be no external (user) visibility that they
>> are
>> made.
>>
>> But at first, I'll start looking at how this may be processed by a
>> standard
>> flex/bison parser. From the work I already did, I know this is not
>> easy, but
>> could probably work.
>>
>> Feedback appreciated.
>>
>> Rainer
>>
>>> -----Original Message-----
>>> From: [email protected] [mailto:rsyslog-
>>> [email protected]] On Behalf Of [email protected]
>>> Sent: Tuesday, July 13, 2010 7:30 PM
>>> To: rsyslog-users
>>> Subject: Re: [rsyslog] NEW rsyslog.conf format
>>>
>>> On Tue, 13 Jul 2010, Rainer Gerhards wrote:
>>>
>>>> To add some of the less obvious things: support for multiple
>>> listeners going
>>>> to different outputs, in an easy to use way. Support for
>> explicitely
>>>> specified concurrency (or serialization) for high-speed
>> environments,
>>> support
>>>> for different queues, and tying of different queues to different
>>> object
>>>> classes (inputs, message processors, actions). Flexibility to
>> support
>>>> configuration for even unexpected plugins that we may not even know
>>> about
>>>> (because some third party writes them and never publishes them).
>>>>
>>>> But I begin to agree that we, the community, as a whole have very
>>> different
>>>> needs. I have gotten the impression that it is probably a good idea
>>> to stop
>>>> the current effort and re-start it with a requirements definition.
>> I
>>> have
>>>> tried to digest the discussions on config format we had over the
>>> year, but
>>>> sometimes it looks like the only consensus we have is that the
>>> current format
>>>> is ugly and hard to use. The solutions are very wide-ranging. I
>> have
>>> even
>>>> briefly looked at syslog-ng, and seen a lot of the pain again that
>>> makes me
>>>> dislike that format (but I'll probably still have a closer look and
>>> will try
>>>> to write up my detailed position why I do not find buying into this
>>> format is
>>>> a good idea).
>>>>
>>>> All in all, I think the best way to re-start our thinking about the
>>> config
>>>> format is to set up a web site where we gather user feedback on
>>>>
>>>> a) what problems they have with the config format
>>>> (not what they just dislike, but real problems, an example
>>>> From myself: it is nearly impossible to get the sequence right
>>>> To bind inputs to the right rulesets AND use ruleset inclusion)
>>>> b) which alternatives they see
>>>>
>>>> With all this being on a web site, it may be possible to craft a
>>> better
>>>> decision in 6 to 9 month, assuming we were able to gain sufficient
>>> feedback
>>>> during that time.
>>>>
>>>> An alternative would be to create a config parser interface, so
>> that
>>>> everybody could code up his favorite config language. However, this
>>> seems to
>>>> be impractical, as many of the ideas floating around (Lua, syslog-
>> ng
>>> style)
>>>> require not just different config parsers, but a different rsyslog
>>> processing
>>>> core as well. Obviously a complete rewrite in the case of Lua, less
>>> for
>>>> syslog-ng, but still considerate. For the syslog-ng style we would
>>> need to
>>>> create named filters, something that I really find questionable.
>>> Also, we
>>>> would need to add an interim layer between inputs and rulesets,
>>> something
>>>> that eats performance. In any case, this are not config-parser only
>>> changes.
>>>>
>>>> Of course, I could just pick one of the alternatives. However, it
>>> doesn't
>>>> make sense to invest a couple of month to do the config system
>>> "right", if we
>>>> know that a lot of folks will still be unhappy after we have done
>>> this.
>>>
>>> one thing that's really good about the current rsyslog config is that
>>> it
>>> makes doing simple things simple, especially for people used to
>> classic
>>> syslog.
>>>
>>> as you say, we need to say what the problems there are with the
>>> existing
>>> config format and look at how to solve those. We don't necessarily
>> need
>>> to
>>> change everything.
>>>
>>> weaknesses that I know of
>>>
>>> inability to easily/clearly define if-then-else
>>>
>>> inability to easily/clearly define/use rulesets
>>>
>>> inability to easily have multiple conditions that go to the same
>>> destination (this can be implemented via rulesets, see above)
>>>
>>> unclear scope rules for config options
>>>
>>> inability to easily/clearly define per-input rules/filters (this is
>>> part
>>> of the scope problem above)
>>>
>>>
>>> I don't think that this requires an entirely new config language. I
>>> think
>>> that almost everything can be solved with a couple simple changes to
>>> the
>>> config language (although as we found when I proposed this on June 25
>>> there are more drastic changes under the covers to
>>> check/correct/document
>>> what gets changed when a config option is processed)
>>>
>>>
>>> Ignoring for the moment the problem of changing how the config
>> options
>>> are
>>> processed (and the fact that you would need to know what data
>>> structures
>>> are managed/created/modified) what does the following proposal _not_
>>> do?
>>>
>>> (pasted in from the mail on June 25 since that's now quite a ways
>> back
>>> in
>>> the archives ;-)
>>>
>>>
>>>
>>> I would propose the following (more or less in order of difficulty)
>>>
>>> introduce scoping
>>>
>>> whenever you see a "{" in the config file, save the current config
>>> options to a stack. when you see a "}" pop the config options from
>> the
>>> stack (undoing any changes in between) introduce statement blocks
>>> change the parser so that wherever it currently allows one action
>>> make
>>> it accept a {} block of actions (treat them as if they were multiple
>>> actions with & as the selector)
>>>
>>> introduce named actions
>>>
>>> something like sub NAME <action> to define and use NAME to use
>>>
>>> introduce if-then-else
>>>
>>> internally you could convert it to
>>>
>>> ruleset {
>>> if condition {
>>> block
>>> discard}
>>> { block } }
>>>
>>> introduct the ability to implcitly define a ruleset
>>>
>>> if an action is a condition (i.e. nested configutations) then
>>> implicitly
>>> create a new ruleset to resolve the nesting.
>>>
>>>
>>> with these capabilities available I think that this will allow for
>>> straightforward config files representing very complex configurations
>>> with
>>> very little change internally to rsyslog.
>>>
>>> I suspect that the result is going to have some bottlenecks in
>> complex
>>> configurations due to all the different rulesets and the passing of
>>> messages between them, but once the capability is in place in the
>>> config
>>> file the implementation under the covers could change later to be
>>> better
>>> optimized.
>>>
>>> as far as the rsyslog config being hard to understand, I think there
>>> are
>>> two conditions here.
>>>
>>> 1. very simple configs where the fact that rsyslog can use a
>>> traditional
>>> syslog config file (with a header on it that seldom needs to change)
>> is
>>> a
>>> huge advantage
>>>
>>> 2. complex configs where the inability to define scope and nest
>> things
>>> is
>>> a major problem and makes the resulting config hard to write.
>>>
>>> David Lang
>>> _______________________________________________
>>> rsyslog mailing list
>>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>>> http://www.rsyslog.com
>> _______________________________________________
>> rsyslog mailing list
>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>> http://www.rsyslog.com
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com
>
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com
