On Fri, 25 Jun 2010, Rainer Gerhards wrote:
-----Original Message-----
From: [email protected] [mailto:rsyslog-
[email protected]] On Behalf Of [email protected]
And as I said: a handler may do mach more. For example the
InputTcpServerListe (or so) directive does not store the port to listen to
but rather starts up the listener -- right at *that point* in config
processing.
this wouldn't be a bad thing with what I was thinking of. the handler
starts up the listener with the variables as they are currently defined.
later on the variables get changed and something else would see the new
versions.
but since there are things changing that you know nothing about, this
won't work
David Lang
In any case, creating a config parser that cleanly handles the existing
config language would be a good thing, and since you are not going to
eliminate the existing config language, it's probably a neccessary
thing
for long-term sanity in any case.
Yes, definitely. And it is a prequisite for
a) cleanly dropping privileges (Ubuntu drops privileges today against my
recommendation and runs into one trouble after another because this is --
documentedly -- not properly supported yet)
b) config reload on HUP (long way to go, but a two-step config read/apply is
a necessary prequiste)
plus maybe some others ;)
Rainer
David Lang
Based on this discussion, I think the proper route is to begin to
structure
config processing and config activation. So that there exist a
parser, one
that creates an in-memory representation of the config, but nothing
else.
Then, in a second step, that in-memory representation is applied and
the
system started up.
This is a non-trivial piece of work. Among others, it requires
considerable
changes to the plugin interface and a rewrite of all plugins. Also,
large
parts of the current config system needs to be replaced. I expect
this takes
until end of summer to complete (but I may be too pessimistic). As a
(re-iterated) side-note, this is the reason I was always so hesitant
to
change the config language.
Once this is done, we will end up with exactly where we are now --
form a
pure users's PoV (no new capabilities, no faster or more reliable
code).
However, we than have a basis where it is relatively simple to
introduce new
config languages or extend the current language.
My original thought was that I could do the redesign together with a
new
config language, so that we would see some immediate benefit.
However, I now
begin to think this is the wrong approach. I should build the basis
first,
then it probably is very easy to define config parsers, which then
only need
to create a proper in-memory representation of the system. I fear,
however,
that I will probably need to change the plugin interface for each new
language, so we should not do this too often ;)
Rainer
-----Original Message-----
From: [email protected] [mailto:rsyslog-
[email protected]] On Behalf Of [email protected]
Sent: Friday, June 25, 2010 7:26 PM
To: rsyslog-users
Subject: Re: [rsyslog] RainerScript revited? NEW rsyslog.conf format
On Fri, 25 Jun 2010, Rainer Gerhards wrote:
Reply to bath David's and Aaron's post:
-----Original Message-----
From: [email protected] [mailto:rsyslog-
[email protected]] On Behalf Of Aaron Wiebe
Sent: Friday, June 25, 2010 5:36 PM
To: rsyslog-users
Subject: Re: [rsyslog] RainerScript revited? NEW rsyslog.conf
format
I agree, we're starting to get to the point of wondering why
again.
So here is my reason why:
My basic concern is scoping. A lot of the current format is
order-specific, implicitly scoped configuration that makes it
extremely hard to work with. With more explicit scoping, I'm
happy.
... that's one reason for the config language.
Next I see a large amount of questions that boil down to the
missing
if/then/else functionality (as elaborated in another post).
The functionality to use a single action in more than one rule is
missing
(but can be emulated using omruleset).
Config-reload on HUP with the current system is impossible (not a
problem of
the config language, but of the engine). Same for full support for
dropping
privileges.
Probably a handful of other, minor things (one being the somewhat
annoying,
always-heard complaint that the config language is hard to work
with,
ugly,
unintuitive, ...).
I would propose the following (more or less in order of difficulty)
introduce scoping
whenever you see a "{" in the config file, save the current
config
options to a stack. when you see a "}" pop the config options from
the
stack (undoing any changes in between)
introduce statement blocks
change the parser so that wherever it currently allows one action
make
it accept a {} block of actions (treat them as if they were multiple
actions with & as the selector)
introduce named actions
something like sub NAME <action> to define and use NAME to use
introduce if-then-else
internally you could convert it to
ruleset {
if condition {
block
discard}
{ block }
}
introduct the ability to implcitly define a ruleset
if an action is a condition (i.e. nested configutations) then
implicitly
create a new ruleset to resolve the nesting.
with these capabilities available I think that this will allow for
straightforward config files representing very complex
configurations
with
very little change internally to rsyslog.
I suspect that the result is going to have some bottlenecks in
complex
configurations due to all the different rulesets and the passing of
messages between them, but once the capability is in place in the
config
file the implementation under the covers could change later to be
better
optimized.
as far as the rsyslog config being hard to understand, I think there
are
two conditions here.
1. very simple configs where the fact that rsyslog can use a
traditional
syslog config file (with a header on it that seldom needs to change)
is
a
huge advantage
2. complex configs where the inability to define scope and nest
things
is
a major problem and makes the resulting config hard to write.
David Lang
But, after all, I managed to use the same config language for years
now. When
Adiscon considered changing its overall config language (probably
going XML
now, but not yet decided), I thought this was a nice chance to get
some
momentum for a new rsyslog config system.
However, if the practical requirement boils down to scope (and it
looks that
way), I am more than happy. All I need to do in that case is change
all
config values to auto-reset after change. That means more config
code
must be
written (copied), but you always know what is active. Probably a
good
week of
work to do (plus a switch to enable the old behavior for
compatibility). I
have no problem with that and could continue to look into some
"more
interesting" things (from my POV, like performance).
And I know this may not be a popular idea, but really, I think it
should be considered -- syslog-ng's format is actually quite
straightforward and easily understood. Having a format similar
may
be
a problem for "product differentiation", but honestly, the syslog-
ng
method of inputs, outputs, and functions to tie those together is
quite appropriate for a logging application. I'm not saying we
should
use their exact format, but I do like lifting some of their better
config ideas. (I'm also partial to the bind-style/syslog-ng-style
config format.)
The config language closely reflects the internal design. If I use
syslog-ng
config, I tie myself into their concepts and restrict me to them.
This is not
something I want to do.
Rainer
-Aaron
On Fri, Jun 25, 2010 at 11:29 AM, <[email protected]> wrote:
On Fri, 25 Jun 2010, Rainer Gerhards wrote:
After all formats seem to have at least some problems attached,
I
decided to
have a look at my original RainerScript idea. Note that the
current
(partial)
implementation is NOT suitable for very high performance.
However,
that is
something that can be changed.
To understand why a language, and why a specialized one, we need
to
look at
rsyslogd from a different angle: to me, rsyslogd is high
performance
processor good at shuffling messages from an input to an output
and
applying
transformations while doing so. There is conditional logic
involved
to craft
the path a specific message takes.
So one can think of rsyslog like a specialized computer whom's
instruction
set is optimized for that purpose (I guess that is what David
was
referring
to when he talked about "functions").
With that view, the configuration file is actually a programming
language:
one that specifies how to process message. Reading the config
file
can then
be compared to a typical compilation, where the generated object
code is
actually targeted towards "rsyslog machine instructions".
Obviously, to program that very special machine, we do need a
specialized
programming language as well. Alternatively, one may think of
this
language
as being the "assembly language" of rsyslog, which closely
matches
the
instruction set and thus can be used to write very efficient
programs (but
requires some skills for non-trivial cases).
Having said this, all language constructs must translate
natively
to
rsyslog
engine objects. At the same time, they should be very simple to
use
for
novice users while providing all the expressiveness an expert
user
needs.
This most importantly means that the config language must be
easy
to
read and
edit by a human.
[Michael: *this* actually are the requirements for the config
language,
together with the other long post this morning -- I am not sure
if
I
find
time to consolidate these two today.]
Now please have a look at a new sample config:
http://download.rsyslog.com/rainerscript_rsyslog.conf
This time, I have not only included a hypothetical complex
configuration, but
also a very simple standard config - just so that we can see how
verbose it
gets. All configs are inside a single file -- you need to scroll
down.
All in all, I have to admit I begin to like the RainerScript
approach once
again. Unfortunately, it obviously is the solution that requires
more work
than any others (because I need to craft the grammar and the
parser
myself).
after reading this I am wondering why everything needs to change.
what concepts are missing in the current config language that you
are
needing to add
as I see it you need to add
ability to define blocks of actions
currently this is done by changing the ruleset, this needs to
be
made
more explicit and obvious
a block of actions may be a single action that's used more than
once
ability to define if-then-else not just if-then
combining these two to make the if-then or if-then-else do blocks
of
actions.
what else is missing?
if you were going to a standard config language that other things
could
understand then changing everything to fit that language makes
sense,
but
if it's going to be another custom config parser what's the
benifit
of
starting from scratch rather than just adding the features to the
existing
one (since you will have to maintain the existing one anyway)
David Lang
P.S. is the rest of adiscon going to just go with XML?
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com
