2015-05-26 13:13 GMT+02:00 singh.janmejay <[email protected]>: > Fixed here: https://github.com/rsyslog/rsyslog/pull/358 > > Some related thoughts: > - this control can be used to exactly count messages > - most load-balancing or sampling scenarios don't need exact counters, > they need to be approximately right > - random-numbers can be used for load-balancing (goes without saying > it doesn't give us counters, if needed for some other reason) > > Global variables work using a global-lock, which is not good for throughput. > > So how about: > - we build support for thread-local (or rather worker-instance-local) counters
trying to fight feature creep: do we *really* need yet another variable class? Aren't we totally confusing users? > - we build a function rand(<max>) which returns a random number 0 <= > result < 'max' sounds like a minimal change (aka "good") to me. Rainer > > I am not suggesting one-of-the-two, instead Im suggesting 'both'. > > On Sat, May 23, 2015 at 8:24 AM, David Lang <[email protected]> wrote: >> you should be able to use global variables instead. no need for a plugin. >> >> David Lang >> >> On Sat, 23 May 2015, chenlin rao wrote: >> >>> Date: Sat, 23 May 2015 10:41:27 +0800 >>> From: chenlin rao <[email protected]> >>> Reply-To: rsyslog-users <[email protected]> >>> To: rsyslog-users <[email protected]> >>> Subject: Re: [rsyslog] cee json + mmsequence >>> >>> >>> mmsequence is deprecated? So which plugin instead? >>> >>> 2015-05-22 21:15 GMT+08:00 singh.janmejay <[email protected]>: >>> >>>> Cool, thats the answer I was looking for. >>>> >>>> >>>> On Fri, May 22, 2015 at 5:55 PM, Rainer Gerhards >>>> <[email protected]> wrote: >>>>> >>>>> 2015-05-21 17:40 GMT+02:00 singh.janmejay <[email protected]>: >>>>>> >>>>>> Are $/ variable manipulations supposed to be atomic? >>>>> >>>>> >>>>> It depends on what you mean with "atomic". The value returned should >>>>> be consistent, that for sure. But if you use >>>>> >>>>> $/var = $/var + 1 >>>> >>>> >>>> Yes, I expect it to jump, but not get corrupted. >>>> >>>>> >>>>> and $/var is 16 when this starts, the result can very well be 18 if >>>>> another thread executed the exact same sequence. I think for this use >>>>> there was an atomic increment function planned. I guess I never >>>>> implemented it, though. >>>>> >>>>>> >>>>>> Eg. $/var % 16 should never produce any number greater than 15, but it >>>> >>>> does. >>>> >>>> This will not be a problem if we use atomic read. >>>> Since you have made it clear that the idea was to make it atomic(which >>>> means this is a bug), I'll fix this. >>>> >>>>> >>>>> as David said, that's a bug. >>>>> >>>>>> >>>>>> Are they non-atomic by design? If so, is mmsequence the right thing to >>>>>> use (assuming it has atomic counters, I haven't checked if it really >>>>>> does)? >>>>> >>>>> >>>>> mmsequence is deprecated. >>>>> >>>>>> >>>>>> There are few options on the table, I guess. >>>>> >>>>> >>>>> before I take a shot at these, let's see if we had the same >>>>> understanding of "atomic". >>>> >>>> >>>> We do. >>>> >>>>> >>>>> Rainer >>>>>> >>>>>> 1. If $/ variables are supposed to be atomic, we just fix the impl. >>>>>> 2. We use mmsequence, and keep it alive (assuming it has atomic >>>> >>>> counters) >>>>>> >>>>>> 3. We fix mmsequence to implement atomic counters. >>>>>> 4. If $/ is not supposed to be atomic (by design), we implement $@ >>>>>> which is thread-local, so people can get atomic counters within thread >>>>>> (evenness in data(for load-balancing or sampling kind of usecase) is >>>>>> maintained by the fact that a single thread will pump a lot of >>>>>> messages, so across all threads it'll approximately be as even as it >>>>>> is in a single-thread. >>>>>> 5. We implement a random-number construct (which i believe either >>>>>> mmsequence or mmcount had) which allows acceptable load-balancing or >>>>>> sampling etc. >>>>>> >>>>>> Thoughts? >>>>>> >>>>>> On Wed, Dec 3, 2014 at 11:12 PM, David Lang <[email protected]> wrote: >>>>>>> >>>>>>> On Wed, 3 Dec 2014, Brian Knox wrote: >>>>>>> >>>>>>>> So two questions then: >>>>>>>> >>>>>>>> 1. In the case of using mmjsonparse on messages where we do not know >>>>>>>> ahead >>>>>>>> of time what all the. json attributes will be, is there a var for a >>>> >>>> place >>>>>>>> >>>>>>>> in the tree where we can get all attributes that were in the cee >>>> >>>> message? >>>>>>> >>>>>>> >>>>>>> >>>>>>> There are three 'classes' of variables >>>>>>> >>>>>>> $! are 'normal' variables, the place that mmjsonparse, etc put the >>>> >>>> results >>>>>>> >>>>>>> of parsing the cee message by default. >>>>>>> >>>>>>> $. are 'local' variables, they are a place you can put stuff that you >>>> >>>> don't >>>>>>> >>>>>>> want to end up in a message when you output $! >>>>>>> >>>>>>> $\ are 'global' variables, unlike the others, they persist from >>>> >>>> message to >>>>>>> >>>>>>> message. >>>>>>> >>>>>>> All of these variables are a tree, where you can have $!foo!bar and >>>>>>> $!foo!baz. If you just reference $!foo it will output both bar and baz >>>> >>>> as >>>>>>> >>>>>>> JSON >>>>>>> >>>>>>> so to see everything, parse the message and then output $! >>>>>>> >>>>>>>> 2. If mmsequence is being deprecated, is there a pointer to how to >>>>>>>> increment global variables to provide a sequence? >>>>>>> >>>>>>> >>>>>>> >>>>>>> set $\var = $\var + 1; >>>>>>> >>>>>>>> 3. This morning I remembered the "mmcount" module - is this also >>>>>>>> being >>>>>>>> deprecated in favor of global variables? >>>>>>> >>>>>>> >>>>>>> >>>>>>> probably. >>>>>>> >>>>>>> David Lang >>>>>>> >>>>>>> >>>>>>>> Brian >>>>>>>> >>>>>>>> On Wed, Dec 3, 2014 at 3:52 AM, Rainer Gerhards < >>>> >>>> [email protected]> >>>>>>>> >>>>>>>> wrote: >>>>>>>> >>>>>>>>> 2014-12-02 21:59 GMT+01:00 Brian Knox <[email protected]>: >>>>>>>>> >>>>>>>>>> We have a case where we are receiving messages in @cee: format, and >>>>>>>>>> would >>>>>>>>>> like to add a sequence variable generated via mmsequence to the >>>> >>>> message. >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> Before I spend too much time digging into this I wanted to ask if, >>>> >>>> using >>>>>>>>>> >>>>>>>>>> some template hackery and json subtrees, this is something that >>>> >>>> sounds >>>>>>>>>> >>>>>>>>>> possible - and if maybe someone else was already doing it. >>>>>>>>>> >>>>>>>>>> >>>>>>>>> David has answered the real question, just let me add that >>>> >>>> mmsequence was >>>>>>>>> >>>>>>>>> a >>>>>>>>> work-around for v7 where global variables were not available. It is >>>> >>>> now >>>>>>>>> >>>>>>>>> deprecated and will probably removed in a couple of month. >>>>>>>>> >>>>>>>>> I'll check the doc if it tells this clearly enough. >>>>>>>>> >>>>>>>>> Rainer >>>>>>>>> >>>>>>>>>> Thanks! >>>>>>>>>> Brian >>>>>>>>>> _______________________________________________ >>>>>>>>>> rsyslog mailing list >>>>>>>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>>>>>>>>> http://www.rsyslog.com/professional-services/ >>>>>>>>>> What's up with rsyslog? Follow https://twitter.com/rgerhards >>>>>>>>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a >>>> >>>> myriad >>>>>>>>>> >>>>>>>>>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if >>>> >>>> you >>>>>>>>>> >>>>>>>>>> DON'T LIKE THAT. >>>>>>>>>> >>>>>>>>> _______________________________________________ >>>>>>>>> rsyslog mailing list >>>>>>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>>>>>>>> http://www.rsyslog.com/professional-services/ >>>>>>>>> What's up with rsyslog? Follow https://twitter.com/rgerhards >>>>>>>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a >>>> >>>> myriad >>>>>>>>> >>>>>>>>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if >>>> >>>> you >>>>>>>>> >>>>>>>>> DON'T LIKE THAT. >>>>>>>>> >>>>>>>> _______________________________________________ >>>>>>>> rsyslog mailing list >>>>>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>>>>>>> http://www.rsyslog.com/professional-services/ >>>>>>>> What's up with rsyslog? Follow https://twitter.com/rgerhards >>>>>>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a >>>> >>>> myriad >>>>>>>> >>>>>>>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if >>>> >>>> you DON'T >>>>>>>> >>>>>>>> LIKE THAT. >>>>>>>> >>>>>>> _______________________________________________ >>>>>>> rsyslog mailing list >>>>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>>>>>> http://www.rsyslog.com/professional-services/ >>>>>>> What's up with rsyslog? Follow https://twitter.com/rgerhards >>>>>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a >>>> >>>> myriad of >>>>>>> >>>>>>> sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you >>>> >>>> DON'T >>>>>>> >>>>>>> LIKE THAT. >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> -- >>>>>> Regards, >>>>>> Janmejay >>>>>> http://codehunk.wordpress.com >>>>>> _______________________________________________ >>>>>> rsyslog mailing list >>>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>>>>> http://www.rsyslog.com/professional-services/ >>>>>> What's up with rsyslog? Follow https://twitter.com/rgerhards >>>>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a >>>> >>>> myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if >>>> you DON'T LIKE THAT. >>>>> >>>>> _______________________________________________ >>>>> rsyslog mailing list >>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>>>> http://www.rsyslog.com/professional-services/ >>>>> What's up with rsyslog? Follow https://twitter.com/rgerhards >>>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad >>>> >>>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you >>>> DON'T LIKE THAT. >>>> >>>> >>>> >>>> -- >>>> Regards, >>>> Janmejay >>>> http://codehunk.wordpress.com >>>> _______________________________________________ >>>> rsyslog mailing list >>>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>>> http://www.rsyslog.com/professional-services/ >>>> What's up with rsyslog? Follow https://twitter.com/rgerhards >>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad >>>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you >>>> DON'T LIKE THAT. >>>> >>> _______________________________________________ >>> rsyslog mailing list >>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>> http://www.rsyslog.com/professional-services/ >>> What's up with rsyslog? Follow https://twitter.com/rgerhards >>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad >>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T >>> LIKE THAT. >>> >> _______________________________________________ >> rsyslog mailing list >> http://lists.adiscon.net/mailman/listinfo/rsyslog >> http://www.rsyslog.com/professional-services/ >> What's up with rsyslog? Follow https://twitter.com/rgerhards >> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of >> sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T >> LIKE THAT. > > > > -- > Regards, > Janmejay > http://codehunk.wordpress.com > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com/professional-services/ > What's up with rsyslog? Follow https://twitter.com/rgerhards > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of > sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T > LIKE THAT. _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
