Fixed here: https://github.com/rsyslog/rsyslog/pull/358
Some related thoughts: - this control can be used to exactly count messages - most load-balancing or sampling scenarios don't need exact counters, they need to be approximately right - random-numbers can be used for load-balancing (goes without saying it doesn't give us counters, if needed for some other reason) Global variables work using a global-lock, which is not good for throughput. So how about: - we build support for thread-local (or rather worker-instance-local) counters - we build a function rand(<max>) which returns a random number 0 <= result < 'max' I am not suggesting one-of-the-two, instead Im suggesting 'both'. On Sat, May 23, 2015 at 8:24 AM, David Lang <[email protected]> wrote: > you should be able to use global variables instead. no need for a plugin. > > David Lang > > On Sat, 23 May 2015, chenlin rao wrote: > >> Date: Sat, 23 May 2015 10:41:27 +0800 >> From: chenlin rao <[email protected]> >> Reply-To: rsyslog-users <[email protected]> >> To: rsyslog-users <[email protected]> >> Subject: Re: [rsyslog] cee json + mmsequence >> >> >> mmsequence is deprecated? So which plugin instead? >> >> 2015-05-22 21:15 GMT+08:00 singh.janmejay <[email protected]>: >> >>> Cool, thats the answer I was looking for. >>> >>> >>> On Fri, May 22, 2015 at 5:55 PM, Rainer Gerhards >>> <[email protected]> wrote: >>>> >>>> 2015-05-21 17:40 GMT+02:00 singh.janmejay <[email protected]>: >>>>> >>>>> Are $/ variable manipulations supposed to be atomic? >>>> >>>> >>>> It depends on what you mean with "atomic". The value returned should >>>> be consistent, that for sure. But if you use >>>> >>>> $/var = $/var + 1 >>> >>> >>> Yes, I expect it to jump, but not get corrupted. >>> >>>> >>>> and $/var is 16 when this starts, the result can very well be 18 if >>>> another thread executed the exact same sequence. I think for this use >>>> there was an atomic increment function planned. I guess I never >>>> implemented it, though. >>>> >>>>> >>>>> Eg. $/var % 16 should never produce any number greater than 15, but it >>> >>> does. >>> >>> This will not be a problem if we use atomic read. >>> Since you have made it clear that the idea was to make it atomic(which >>> means this is a bug), I'll fix this. >>> >>>> >>>> as David said, that's a bug. >>>> >>>>> >>>>> Are they non-atomic by design? If so, is mmsequence the right thing to >>>>> use (assuming it has atomic counters, I haven't checked if it really >>>>> does)? >>>> >>>> >>>> mmsequence is deprecated. >>>> >>>>> >>>>> There are few options on the table, I guess. >>>> >>>> >>>> before I take a shot at these, let's see if we had the same >>>> understanding of "atomic". >>> >>> >>> We do. >>> >>>> >>>> Rainer >>>>> >>>>> 1. If $/ variables are supposed to be atomic, we just fix the impl. >>>>> 2. We use mmsequence, and keep it alive (assuming it has atomic >>> >>> counters) >>>>> >>>>> 3. We fix mmsequence to implement atomic counters. >>>>> 4. If $/ is not supposed to be atomic (by design), we implement $@ >>>>> which is thread-local, so people can get atomic counters within thread >>>>> (evenness in data(for load-balancing or sampling kind of usecase) is >>>>> maintained by the fact that a single thread will pump a lot of >>>>> messages, so across all threads it'll approximately be as even as it >>>>> is in a single-thread. >>>>> 5. We implement a random-number construct (which i believe either >>>>> mmsequence or mmcount had) which allows acceptable load-balancing or >>>>> sampling etc. >>>>> >>>>> Thoughts? >>>>> >>>>> On Wed, Dec 3, 2014 at 11:12 PM, David Lang <[email protected]> wrote: >>>>>> >>>>>> On Wed, 3 Dec 2014, Brian Knox wrote: >>>>>> >>>>>>> So two questions then: >>>>>>> >>>>>>> 1. In the case of using mmjsonparse on messages where we do not know >>>>>>> ahead >>>>>>> of time what all the. json attributes will be, is there a var for a >>> >>> place >>>>>>> >>>>>>> in the tree where we can get all attributes that were in the cee >>> >>> message? >>>>>> >>>>>> >>>>>> >>>>>> There are three 'classes' of variables >>>>>> >>>>>> $! are 'normal' variables, the place that mmjsonparse, etc put the >>> >>> results >>>>>> >>>>>> of parsing the cee message by default. >>>>>> >>>>>> $. are 'local' variables, they are a place you can put stuff that you >>> >>> don't >>>>>> >>>>>> want to end up in a message when you output $! >>>>>> >>>>>> $\ are 'global' variables, unlike the others, they persist from >>> >>> message to >>>>>> >>>>>> message. >>>>>> >>>>>> All of these variables are a tree, where you can have $!foo!bar and >>>>>> $!foo!baz. If you just reference $!foo it will output both bar and baz >>> >>> as >>>>>> >>>>>> JSON >>>>>> >>>>>> so to see everything, parse the message and then output $! >>>>>> >>>>>>> 2. If mmsequence is being deprecated, is there a pointer to how to >>>>>>> increment global variables to provide a sequence? >>>>>> >>>>>> >>>>>> >>>>>> set $\var = $\var + 1; >>>>>> >>>>>>> 3. This morning I remembered the "mmcount" module - is this also >>>>>>> being >>>>>>> deprecated in favor of global variables? >>>>>> >>>>>> >>>>>> >>>>>> probably. >>>>>> >>>>>> David Lang >>>>>> >>>>>> >>>>>>> Brian >>>>>>> >>>>>>> On Wed, Dec 3, 2014 at 3:52 AM, Rainer Gerhards < >>> >>> [email protected]> >>>>>>> >>>>>>> wrote: >>>>>>> >>>>>>>> 2014-12-02 21:59 GMT+01:00 Brian Knox <[email protected]>: >>>>>>>> >>>>>>>>> We have a case where we are receiving messages in @cee: format, and >>>>>>>>> would >>>>>>>>> like to add a sequence variable generated via mmsequence to the >>> >>> message. >>>>>>>>> >>>>>>>>> >>>>>>>>> Before I spend too much time digging into this I wanted to ask if, >>> >>> using >>>>>>>>> >>>>>>>>> some template hackery and json subtrees, this is something that >>> >>> sounds >>>>>>>>> >>>>>>>>> possible - and if maybe someone else was already doing it. >>>>>>>>> >>>>>>>>> >>>>>>>> David has answered the real question, just let me add that >>> >>> mmsequence was >>>>>>>> >>>>>>>> a >>>>>>>> work-around for v7 where global variables were not available. It is >>> >>> now >>>>>>>> >>>>>>>> deprecated and will probably removed in a couple of month. >>>>>>>> >>>>>>>> I'll check the doc if it tells this clearly enough. >>>>>>>> >>>>>>>> Rainer >>>>>>>> >>>>>>>>> Thanks! >>>>>>>>> Brian >>>>>>>>> _______________________________________________ >>>>>>>>> rsyslog mailing list >>>>>>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>>>>>>>> http://www.rsyslog.com/professional-services/ >>>>>>>>> What's up with rsyslog? Follow https://twitter.com/rgerhards >>>>>>>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a >>> >>> myriad >>>>>>>>> >>>>>>>>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if >>> >>> you >>>>>>>>> >>>>>>>>> DON'T LIKE THAT. >>>>>>>>> >>>>>>>> _______________________________________________ >>>>>>>> rsyslog mailing list >>>>>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>>>>>>> http://www.rsyslog.com/professional-services/ >>>>>>>> What's up with rsyslog? Follow https://twitter.com/rgerhards >>>>>>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a >>> >>> myriad >>>>>>>> >>>>>>>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if >>> >>> you >>>>>>>> >>>>>>>> DON'T LIKE THAT. >>>>>>>> >>>>>>> _______________________________________________ >>>>>>> rsyslog mailing list >>>>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>>>>>> http://www.rsyslog.com/professional-services/ >>>>>>> What's up with rsyslog? Follow https://twitter.com/rgerhards >>>>>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a >>> >>> myriad >>>>>>> >>>>>>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if >>> >>> you DON'T >>>>>>> >>>>>>> LIKE THAT. >>>>>>> >>>>>> _______________________________________________ >>>>>> rsyslog mailing list >>>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>>>>> http://www.rsyslog.com/professional-services/ >>>>>> What's up with rsyslog? Follow https://twitter.com/rgerhards >>>>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a >>> >>> myriad of >>>>>> >>>>>> sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you >>> >>> DON'T >>>>>> >>>>>> LIKE THAT. >>>>> >>>>> >>>>> >>>>> >>>>> -- >>>>> Regards, >>>>> Janmejay >>>>> http://codehunk.wordpress.com >>>>> _______________________________________________ >>>>> rsyslog mailing list >>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>>>> http://www.rsyslog.com/professional-services/ >>>>> What's up with rsyslog? Follow https://twitter.com/rgerhards >>>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a >>> >>> myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if >>> you DON'T LIKE THAT. >>>> >>>> _______________________________________________ >>>> rsyslog mailing list >>>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>>> http://www.rsyslog.com/professional-services/ >>>> What's up with rsyslog? Follow https://twitter.com/rgerhards >>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad >>> >>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you >>> DON'T LIKE THAT. >>> >>> >>> >>> -- >>> Regards, >>> Janmejay >>> http://codehunk.wordpress.com >>> _______________________________________________ >>> rsyslog mailing list >>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>> http://www.rsyslog.com/professional-services/ >>> What's up with rsyslog? Follow https://twitter.com/rgerhards >>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad >>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you >>> DON'T LIKE THAT. >>> >> _______________________________________________ >> rsyslog mailing list >> http://lists.adiscon.net/mailman/listinfo/rsyslog >> http://www.rsyslog.com/professional-services/ >> What's up with rsyslog? Follow https://twitter.com/rgerhards >> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad >> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T >> LIKE THAT. >> > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com/professional-services/ > What's up with rsyslog? Follow https://twitter.com/rgerhards > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of > sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T > LIKE THAT. -- Regards, Janmejay http://codehunk.wordpress.com _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
