Ok, rand(max) it is then. -- Regards, Janmejay
PS: Please blame the typos in this mail on my phone's uncivilized soft keyboard sporting it's not-so-smart-assist technology. On May 26, 2015 6:59 PM, "Rainer Gerhards" <[email protected]> wrote: > 2015-05-26 13:13 GMT+02:00 singh.janmejay <[email protected]>: > > Fixed here: https://github.com/rsyslog/rsyslog/pull/358 > > > > Some related thoughts: > > - this control can be used to exactly count messages > > - most load-balancing or sampling scenarios don't need exact counters, > > they need to be approximately right > > - random-numbers can be used for load-balancing (goes without saying > > it doesn't give us counters, if needed for some other reason) > > > > Global variables work using a global-lock, which is not good for > throughput. > > > > So how about: > > - we build support for thread-local (or rather worker-instance-local) > counters > > trying to fight feature creep: do we *really* need yet another > variable class? Aren't we totally confusing users? > > > - we build a function rand(<max>) which returns a random number 0 <= > > result < 'max' > > sounds like a minimal change (aka "good") to me. > > Rainer > > > > I am not suggesting one-of-the-two, instead Im suggesting 'both'. > > > > On Sat, May 23, 2015 at 8:24 AM, David Lang <[email protected]> wrote: > >> you should be able to use global variables instead. no need for a > plugin. > >> > >> David Lang > >> > >> On Sat, 23 May 2015, chenlin rao wrote: > >> > >>> Date: Sat, 23 May 2015 10:41:27 +0800 > >>> From: chenlin rao <[email protected]> > >>> Reply-To: rsyslog-users <[email protected]> > >>> To: rsyslog-users <[email protected]> > >>> Subject: Re: [rsyslog] cee json + mmsequence > >>> > >>> > >>> mmsequence is deprecated? So which plugin instead? > >>> > >>> 2015-05-22 21:15 GMT+08:00 singh.janmejay <[email protected]>: > >>> > >>>> Cool, thats the answer I was looking for. > >>>> > >>>> > >>>> On Fri, May 22, 2015 at 5:55 PM, Rainer Gerhards > >>>> <[email protected]> wrote: > >>>>> > >>>>> 2015-05-21 17:40 GMT+02:00 singh.janmejay <[email protected] > >: > >>>>>> > >>>>>> Are $/ variable manipulations supposed to be atomic? > >>>>> > >>>>> > >>>>> It depends on what you mean with "atomic". The value returned should > >>>>> be consistent, that for sure. But if you use > >>>>> > >>>>> $/var = $/var + 1 > >>>> > >>>> > >>>> Yes, I expect it to jump, but not get corrupted. > >>>> > >>>>> > >>>>> and $/var is 16 when this starts, the result can very well be 18 if > >>>>> another thread executed the exact same sequence. I think for this use > >>>>> there was an atomic increment function planned. I guess I never > >>>>> implemented it, though. > >>>>> > >>>>>> > >>>>>> Eg. $/var % 16 should never produce any number greater than 15, but > it > >>>> > >>>> does. > >>>> > >>>> This will not be a problem if we use atomic read. > >>>> Since you have made it clear that the idea was to make it atomic(which > >>>> means this is a bug), I'll fix this. > >>>> > >>>>> > >>>>> as David said, that's a bug. > >>>>> > >>>>>> > >>>>>> Are they non-atomic by design? If so, is mmsequence the right thing > to > >>>>>> use (assuming it has atomic counters, I haven't checked if it really > >>>>>> does)? > >>>>> > >>>>> > >>>>> mmsequence is deprecated. > >>>>> > >>>>>> > >>>>>> There are few options on the table, I guess. > >>>>> > >>>>> > >>>>> before I take a shot at these, let's see if we had the same > >>>>> understanding of "atomic". > >>>> > >>>> > >>>> We do. > >>>> > >>>>> > >>>>> Rainer > >>>>>> > >>>>>> 1. If $/ variables are supposed to be atomic, we just fix the impl. > >>>>>> 2. We use mmsequence, and keep it alive (assuming it has atomic > >>>> > >>>> counters) > >>>>>> > >>>>>> 3. We fix mmsequence to implement atomic counters. > >>>>>> 4. If $/ is not supposed to be atomic (by design), we implement $@ > >>>>>> which is thread-local, so people can get atomic counters within > thread > >>>>>> (evenness in data(for load-balancing or sampling kind of usecase) is > >>>>>> maintained by the fact that a single thread will pump a lot of > >>>>>> messages, so across all threads it'll approximately be as even as it > >>>>>> is in a single-thread. > >>>>>> 5. We implement a random-number construct (which i believe either > >>>>>> mmsequence or mmcount had) which allows acceptable load-balancing or > >>>>>> sampling etc. > >>>>>> > >>>>>> Thoughts? > >>>>>> > >>>>>> On Wed, Dec 3, 2014 at 11:12 PM, David Lang <[email protected]> wrote: > >>>>>>> > >>>>>>> On Wed, 3 Dec 2014, Brian Knox wrote: > >>>>>>> > >>>>>>>> So two questions then: > >>>>>>>> > >>>>>>>> 1. In the case of using mmjsonparse on messages where we do not > know > >>>>>>>> ahead > >>>>>>>> of time what all the. json attributes will be, is there a var for > a > >>>> > >>>> place > >>>>>>>> > >>>>>>>> in the tree where we can get all attributes that were in the cee > >>>> > >>>> message? > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> There are three 'classes' of variables > >>>>>>> > >>>>>>> $! are 'normal' variables, the place that mmjsonparse, etc put the > >>>> > >>>> results > >>>>>>> > >>>>>>> of parsing the cee message by default. > >>>>>>> > >>>>>>> $. are 'local' variables, they are a place you can put stuff that > you > >>>> > >>>> don't > >>>>>>> > >>>>>>> want to end up in a message when you output $! > >>>>>>> > >>>>>>> $\ are 'global' variables, unlike the others, they persist from > >>>> > >>>> message to > >>>>>>> > >>>>>>> message. > >>>>>>> > >>>>>>> All of these variables are a tree, where you can have $!foo!bar and > >>>>>>> $!foo!baz. If you just reference $!foo it will output both bar and > baz > >>>> > >>>> as > >>>>>>> > >>>>>>> JSON > >>>>>>> > >>>>>>> so to see everything, parse the message and then output $! > >>>>>>> > >>>>>>>> 2. If mmsequence is being deprecated, is there a pointer to how to > >>>>>>>> increment global variables to provide a sequence? > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> set $\var = $\var + 1; > >>>>>>> > >>>>>>>> 3. This morning I remembered the "mmcount" module - is this also > >>>>>>>> being > >>>>>>>> deprecated in favor of global variables? > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> probably. > >>>>>>> > >>>>>>> David Lang > >>>>>>> > >>>>>>> > >>>>>>>> Brian > >>>>>>>> > >>>>>>>> On Wed, Dec 3, 2014 at 3:52 AM, Rainer Gerhards < > >>>> > >>>> [email protected]> > >>>>>>>> > >>>>>>>> wrote: > >>>>>>>> > >>>>>>>>> 2014-12-02 21:59 GMT+01:00 Brian Knox <[email protected]>: > >>>>>>>>> > >>>>>>>>>> We have a case where we are receiving messages in @cee: format, > and > >>>>>>>>>> would > >>>>>>>>>> like to add a sequence variable generated via mmsequence to the > >>>> > >>>> message. > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> Before I spend too much time digging into this I wanted to ask > if, > >>>> > >>>> using > >>>>>>>>>> > >>>>>>>>>> some template hackery and json subtrees, this is something that > >>>> > >>>> sounds > >>>>>>>>>> > >>>>>>>>>> possible - and if maybe someone else was already doing it. > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>> David has answered the real question, just let me add that > >>>> > >>>> mmsequence was > >>>>>>>>> > >>>>>>>>> a > >>>>>>>>> work-around for v7 where global variables were not available. It > is > >>>> > >>>> now > >>>>>>>>> > >>>>>>>>> deprecated and will probably removed in a couple of month. > >>>>>>>>> > >>>>>>>>> I'll check the doc if it tells this clearly enough. > >>>>>>>>> > >>>>>>>>> Rainer > >>>>>>>>> > >>>>>>>>>> Thanks! > >>>>>>>>>> Brian > >>>>>>>>>> _______________________________________________ > >>>>>>>>>> rsyslog mailing list > >>>>>>>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog > >>>>>>>>>> http://www.rsyslog.com/professional-services/ > >>>>>>>>>> What's up with rsyslog? Follow https://twitter.com/rgerhards > >>>>>>>>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by > a > >>>> > >>>> myriad > >>>>>>>>>> > >>>>>>>>>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST > if > >>>> > >>>> you > >>>>>>>>>> > >>>>>>>>>> DON'T LIKE THAT. > >>>>>>>>>> > >>>>>>>>> _______________________________________________ > >>>>>>>>> rsyslog mailing list > >>>>>>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog > >>>>>>>>> http://www.rsyslog.com/professional-services/ > >>>>>>>>> What's up with rsyslog? Follow https://twitter.com/rgerhards > >>>>>>>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a > >>>> > >>>> myriad > >>>>>>>>> > >>>>>>>>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST > if > >>>> > >>>> you > >>>>>>>>> > >>>>>>>>> DON'T LIKE THAT. > >>>>>>>>> > >>>>>>>> _______________________________________________ > >>>>>>>> rsyslog mailing list > >>>>>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog > >>>>>>>> http://www.rsyslog.com/professional-services/ > >>>>>>>> What's up with rsyslog? Follow https://twitter.com/rgerhards > >>>>>>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a > >>>> > >>>> myriad > >>>>>>>> > >>>>>>>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if > >>>> > >>>> you DON'T > >>>>>>>> > >>>>>>>> LIKE THAT. > >>>>>>>> > >>>>>>> _______________________________________________ > >>>>>>> rsyslog mailing list > >>>>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog > >>>>>>> http://www.rsyslog.com/professional-services/ > >>>>>>> What's up with rsyslog? Follow https://twitter.com/rgerhards > >>>>>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a > >>>> > >>>> myriad of > >>>>>>> > >>>>>>> sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you > >>>> > >>>> DON'T > >>>>>>> > >>>>>>> LIKE THAT. > >>>>>> > >>>>>> > >>>>>> > >>>>>> > >>>>>> -- > >>>>>> Regards, > >>>>>> Janmejay > >>>>>> http://codehunk.wordpress.com > >>>>>> _______________________________________________ > >>>>>> rsyslog mailing list > >>>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog > >>>>>> http://www.rsyslog.com/professional-services/ > >>>>>> What's up with rsyslog? Follow https://twitter.com/rgerhards > >>>>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a > >>>> > >>>> myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT > POST if > >>>> you DON'T LIKE THAT. > >>>>> > >>>>> _______________________________________________ > >>>>> rsyslog mailing list > >>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog > >>>>> http://www.rsyslog.com/professional-services/ > >>>>> What's up with rsyslog? Follow https://twitter.com/rgerhards > >>>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a > myriad > >>>> > >>>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you > >>>> DON'T LIKE THAT. > >>>> > >>>> > >>>> > >>>> -- > >>>> Regards, > >>>> Janmejay > >>>> http://codehunk.wordpress.com > >>>> _______________________________________________ > >>>> rsyslog mailing list > >>>> http://lists.adiscon.net/mailman/listinfo/rsyslog > >>>> http://www.rsyslog.com/professional-services/ > >>>> What's up with rsyslog? Follow https://twitter.com/rgerhards > >>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a > myriad > >>>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you > >>>> DON'T LIKE THAT. > >>>> > >>> _______________________________________________ > >>> rsyslog mailing list > >>> http://lists.adiscon.net/mailman/listinfo/rsyslog > >>> http://www.rsyslog.com/professional-services/ > >>> What's up with rsyslog? Follow https://twitter.com/rgerhards > >>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a > myriad > >>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you > DON'T > >>> LIKE THAT. > >>> > >> _______________________________________________ > >> rsyslog mailing list > >> http://lists.adiscon.net/mailman/listinfo/rsyslog > >> http://www.rsyslog.com/professional-services/ > >> What's up with rsyslog? Follow https://twitter.com/rgerhards > >> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a > myriad of > >> sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you > DON'T > >> LIKE THAT. > > > > > > > > -- > > Regards, > > Janmejay > > http://codehunk.wordpress.com > > _______________________________________________ > > rsyslog mailing list > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > http://www.rsyslog.com/professional-services/ > > What's up with rsyslog? Follow https://twitter.com/rgerhards > > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad > of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you > DON'T LIKE THAT. > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com/professional-services/ > What's up with rsyslog? Follow https://twitter.com/rgerhards > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad > of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you > DON'T LIKE THAT. > _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
