On Tue, 22 Oct 2013, Rainer Gerhards wrote:
On Tue, Oct 22, 2013 at 10:23 AM, David Lang <[email protected]> wrote:
On Tue, 22 Oct 2013, Rainer Gerhards wrote:
Hi all,
I wonder what are the actual use cases behind the desire to have global
variables in rsyslog. There are two that I know (I describe them very
briefly as they were expressed at length in the past couple of days):
1. counting messages for some reason
2. load-balancing output actions
Number 2 is - I think - a weak one, as this should better go into a
specific facility inside the engine (read: nothing that the admin should
need to emulate in the conf file).
Are there any other *real-world* use cases?
I am asking because I would like to think if future development directions
of the rsyslog engine should consider the re-addition of global variable
support (which requires a major rewrite).
well, the original request that lead to getting global variables added was
to have the contents of one message set a variable that could be used in
processing future messages.
I thought that original use case was just counting messages?
What you describe sounds like a correlation use case. Is this something
that we want to do in rsyslog?
For this use, the current functionality works as-is.
Not really. It works ONLY if you set batch size to 1, which is both
counter-intuitive and not something you really want to do.
Sample why it doesn't work:
conf in pseudo-code:
step 1: write content of $/globvar to file
step 2: set $/globvar = $msgprop
Let's say we have 4 messages, each containing a field (msgprop) with
content A, B, C, D. You want that the successsive message contains the
field from the previous one (starting with blank for the first one). So you
would probably expect to see
1: blank
2: A
3: B
4: C
However, the rule engine would evalute the write in step 1, without even
setting globvar at all. So this would lleading to the actual result of
1: blank
2: blank
3: blank
4: blank
In the next batch, all messages would contain "D" and so on...
Most folks would consider this to be an error ;) I agree to Pavel that this
is so far from being what one expects, that even documenting how things
work will not really help getting away from that error-like feeling. Add
the fact that the result is non-deterministic as it depends on the current
batch size, I would also conclude that this is errornous *design*.
I probably was not clear enoug in my initial question: as I said yesterday
evening, I have now removed global variable support. It won't be available
in 7.5.6. My question thus was on the use cases that could guide us if
working toward a correct implementation makes any sense. This would require
a big design change.
the use case wasn't trying to be something as precise as you are talking about,
but rather a much corser ability that after you see message X show up, start
doing something different. This could be to write files to a different location
because a failover has happened and you are now the active box, it could be that
if a particular type of log message shows up you want to log debug messages, or
something else.
given that logs can be re-ordered by the network, by relays in the middle, or by
the results of batch processing, it was never going to be an exact thing, so if
a change only takes effect for the next batch, that's not unreasonable.
yes, this is edging towards log correlation territory, but it's an extremely
handy way to be able to turn things on and off, or to cause logs to be handled
differently without requiring on-the-fly editing of config files and a complete
restart.
the log losses from a complete restart are massive compared to the global
variable impresision of only setting the variable on the next batch.
global variables are not that useful as something that you would set for each
message that you process, but if they are only occasionally changed, they can be
very useful and not a significant performance issue.
However, I do question about variables only being set once per batch, if you are
processing a bunch of messages, and trying to set a variable that you will then
use in templates, which message gets used?
for example, if I want to:
set $!trusted!source = $fromhost-ip;
which message in a batch sets this variable?
were the global variables handled differently than normal or local variables?
David Lang
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
