On Tue, Oct 22, 2013 at 11:09 AM, David Lang <[email protected]> wrote: > On Tue, 22 Oct 2013, Rainer Gerhards wrote: > > On Tue, Oct 22, 2013 at 10:23 AM, David Lang <[email protected]> wrote: >> >> On Tue, 22 Oct 2013, Rainer Gerhards wrote: >>> >>> Hi all, >>> >>>> >>>> I wonder what are the actual use cases behind the desire to have global >>>> variables in rsyslog. There are two that I know (I describe them very >>>> briefly as they were expressed at length in the past couple of days): >>>> >>>> 1. counting messages for some reason >>>> 2. load-balancing output actions >>>> >>>> Number 2 is - I think - a weak one, as this should better go into a >>>> specific facility inside the engine (read: nothing that the admin should >>>> need to emulate in the conf file). >>>> >>>> Are there any other *real-world* use cases? >>>> >>>> I am asking because I would like to think if future development >>>> directions >>>> of the rsyslog engine should consider the re-addition of global variable >>>> support (which requires a major rewrite). >>>> >>>> >>> well, the original request that lead to getting global variables added >>> was >>> to have the contents of one message set a variable that could be used in >>> processing future messages. >>> >>> >>> I thought that original use case was just counting messages? >> >> What you describe sounds like a correlation use case. Is this something >> that we want to do in rsyslog? >> >> For this use, the current functionality works as-is. >>> >>> >>> Not really. It works ONLY if you set batch size to 1, which is both >> counter-intuitive and not something you really want to do. >> >> Sample why it doesn't work: >> >> conf in pseudo-code: >> >> step 1: write content of $/globvar to file >> step 2: set $/globvar = $msgprop >> >> Let's say we have 4 messages, each containing a field (msgprop) with >> content A, B, C, D. You want that the successsive message contains the >> field from the previous one (starting with blank for the first one). So >> you >> would probably expect to see >> >> 1: blank >> 2: A >> 3: B >> 4: C >> >> However, the rule engine would evalute the write in step 1, without even >> setting globvar at all. So this would lleading to the actual result of >> >> 1: blank >> 2: blank >> 3: blank >> 4: blank >> >> In the next batch, all messages would contain "D" and so on... >> >> Most folks would consider this to be an error ;) I agree to Pavel that >> this >> is so far from being what one expects, that even documenting how things >> work will not really help getting away from that error-like feeling. Add >> the fact that the result is non-deterministic as it depends on the current >> batch size, I would also conclude that this is errornous *design*. >> >> I probably was not clear enoug in my initial question: as I said yesterday >> evening, I have now removed global variable support. It won't be available >> in 7.5.6. My question thus was on the use cases that could guide us if >> working toward a correct implementation makes any sense. This would >> require >> a big design change. >> > > the use case wasn't trying to be something as precise as you are talking > about, but rather a much corser ability that after you see message X show > up, start doing something different. This could be to write files to a > different location because a failover has happened and you are now the > active box, it could be that if a particular type of log message shows up > you want to log debug messages, or something else. > > given that logs can be re-ordered by the network, by relays in the middle, > or by the results of batch processing, it was never going to be an exact > thing, so if a change only takes effect for the next batch, that's not > unreasonable. > > yes, this is edging towards log correlation territory, but it's an > extremely handy way to be able to turn things on and off, or to cause logs > to be handled differently without requiring on-the-fly editing of config > files and a complete restart. > > the log losses from a complete restart are massive compared to the global > variable impresision of only setting the variable on the next batch. > > > > global variables are not that useful as something that you would set for > each message that you process, but if they are only occasionally changed, > they can be very useful and not a significant performance issue. > > It's not the performance issue -- it's the "strong smell of incorrectness". Remember that I must support that whole thing in the future. People will continously ask why it "does not work", and some of them will (even/hopefully/probably) have support contracts, so I am actually in charge to deliver a solution. If I then just describe "it is as it is" that will create very unsatisfied customers. So better do it right or do it not at all (which is what I currently opt for, and my peers will kill me if I knowingly introduce a feature that potentially creates great user-desatisfaction).
> However, I do question about variables only being set once per batch, > no, no, it's set once per message! but in the case of global vars, only the last one survices. Take the example from above. The engine executes for all messages in batch: do write action for all messages in batch: do assgin action If we look at step 2, this means (if we unroll the loop): set global = A set global = B set global = C set global = D obviously, only the last value survives. The problem is exclusive to global variables, and the core reason is that the engine was never design to support global scope. I unfortunately forgot about this when I tried to rush this "feature" in quickly before I left for vacation. At that time I should have said "it is not possible with the current engine". The engine is designed to work on message scope, and as long as all is done in that scope, everything is fine. Along the same lines, that's one reason why I always say correlation is not something that rsyslog is made for. Rainer if you are processing a bunch of messages, and trying to set a variable > that you will then use in templates, which message gets used? > > for example, if I want to: > > set $!trusted!source = $fromhost-ip; > > which message in a batch sets this variable? > > were the global variables handled differently than normal or local > variables? > > > > > > David Lang > ______________________________**_________________ > rsyslog mailing list > http://lists.adiscon.net/**mailman/listinfo/rsyslog<http://lists.adiscon.net/mailman/listinfo/rsyslog> > http://www.rsyslog.com/**professional-services/<http://www.rsyslog.com/professional-services/> > What's up with rsyslog? Follow https://twitter.com/rgerhards > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad > of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you > DON'T LIKE THAT. > _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
