On Mon, Oct 21, 2013 at 12:16 PM, David Lang <[email protected]> wrote: > On Mon, 21 Oct 2013, Rainer Gerhards wrote: > > On Mon, Oct 21, 2013 at 10:42 AM, Rainer Gerhards >> <[email protected]>**wrote: >> >> On Mon, Oct 21, 2013 at 8:52 AM, Rainer Gerhards < >>> [email protected] >>> >>>> wrote: >>>> >>> >>> On Sun, Oct 20, 2013 at 6:03 PM, Pavel Levshin <[email protected] >>>> >wrote: >>>> >>>> >>>>> I am unable to reproduce this behaviour with global variables. This is >>>>> what I've tried among others: >>>>> >>>>> if $/zz % 3 == 0 or $/zz % 3 == 1 then { >>>>> set $/zz = $/zz + 1; >>>>> action(...) >>>>> } else { >>>>> set $/zz = 0; >>>>> action(...) >>>>> } >>>>> >>>>> Could you please explain how is it supposed to work? >>>>> >>>>> >>>> It's supposed to work just as you describe it. But indeed, it doesn't do >>>> so, I can reproduce the problem. Looks like a regression. Thanks for >>>> reporting, will now look into it. >>>> >>>> >>> OK, looks like I stumbled into my own trap. In script, you access >>> properties via $<propname>. Global variables have the name $/zz (with zz >>> being the real name). So to access them, you need to access $$/zz. >>> >>> I think I got confused about this some time ago, and the doc is also not >>> correct or at least inconsistent. I now need to work my way through that >>> mess. Just thought I give you some explanation. >>> >>> >> ... and it gets even worse: when I quickly implemented global vars before >> my vacation, I did not fully think out the implications :-( >> >> The use case then was to have some incrementing counters. That works. >> However, due to the way the rule engine works, this use case here does not >> work as expected. The reason is that the rule engine is a SIMD machine: it >> applies all operations to all message inside the batch. As such, it does >> the if-evaluation on the whole batch, then the variable increment, than >> the >> action and so on. It works this way because this is a very big performance >> gain. However, this means when zz is 0 at start of a batch of 10 messages, >> the if will work with 0 for all 10 of them. Then, zz will be ten times >> incremented, leding to value 10. Then, the action is executed (10 times). >> >> I am not sure what the proper solution is. The most clean solution would >> be >> to remove support for global variables, the best but very costly would be >> to enhance the engine to have a special execution mode (much slower) for >> statements that involve global variables. The quick and dirty solution is >> to leave everything as is- but document this behaviour -- as there are >> valid use cases for the current state. >> >> I think I will take the "just document" path for now, and think about >> removing global variables (after all, so far they are only in the >> devel/experimental tree). Comments are appreciated. >> >> @Paval: did you check that mmsequence actually works as you expect? IMHO >> it should expose the same behaviour as global variables. >> > > I think it's fine to just document for now. > > I think the 'right' answer is to create a magic variable that is the > number of messages in the batch. This will allow you to have a counter of > the number of messages. > > now, the counter will not give every message a unique value if you have > batch size > 1, but this is actually pretty much the case with other things > related to the batch (like timestamps on received messages), so it's not > unusual. > > As far as load balancing goes, balancing a batch of messages at a time is > actually 'good enough', and since it saves you manipulating individual > messages, it's a performance win as well. >
I think this variable shuffling is the right approach to load balancing. As I wrote yesterday, I think it is fairly easy to do it inside the engine. If I had not been lost in the vars, I would have started with this today and were positive I could have finished it quickly. If the remaining time permits, I'll probably still try to squeeze it in. > > Since this can be dealt with by just setting the batch size = 1, I think > that it's acceptable. > yup, except for performance (the ES http request with batch size 1 ;)). But your are basically right for most things. Rainer > > David Lang > > ______________________________**_________________ > rsyslog mailing list > http://lists.adiscon.net/**mailman/listinfo/rsyslog<http://lists.adiscon.net/mailman/listinfo/rsyslog> > http://www.rsyslog.com/**professional-services/<http://www.rsyslog.com/professional-services/> > What's up with rsyslog? Follow https://twitter.com/rgerhards > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad > of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you > DON'T LIKE THAT. > _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
