Sorry for the delay, here it is: https://github.com/rsyslog/rsyslog/pull/410
On Tue, May 26, 2015 at 7:20 PM, singh.janmejay <[email protected]> wrote: > Ok, rand(max) it is then. > > -- > Regards, > Janmejay > > PS: Please blame the typos in this mail on my phone's uncivilized soft > keyboard sporting it's not-so-smart-assist technology. > > > On May 26, 2015 6:59 PM, "Rainer Gerhards" <[email protected]> wrote: >> >> 2015-05-26 13:13 GMT+02:00 singh.janmejay <[email protected]>: >> > Fixed here: https://github.com/rsyslog/rsyslog/pull/358 >> > >> > Some related thoughts: >> > - this control can be used to exactly count messages >> > - most load-balancing or sampling scenarios don't need exact counters, >> > they need to be approximately right >> > - random-numbers can be used for load-balancing (goes without saying >> > it doesn't give us counters, if needed for some other reason) >> > >> > Global variables work using a global-lock, which is not good for >> > throughput. >> > >> > So how about: >> > - we build support for thread-local (or rather worker-instance-local) >> > counters >> >> trying to fight feature creep: do we *really* need yet another >> variable class? Aren't we totally confusing users? >> >> > - we build a function rand(<max>) which returns a random number 0 <= >> > result < 'max' >> >> sounds like a minimal change (aka "good") to me. >> >> Rainer >> > >> > I am not suggesting one-of-the-two, instead Im suggesting 'both'. >> > >> > On Sat, May 23, 2015 at 8:24 AM, David Lang <[email protected]> wrote: >> >> you should be able to use global variables instead. no need for a >> >> plugin. >> >> >> >> David Lang >> >> >> >> On Sat, 23 May 2015, chenlin rao wrote: >> >> >> >>> Date: Sat, 23 May 2015 10:41:27 +0800 >> >>> From: chenlin rao <[email protected]> >> >>> Reply-To: rsyslog-users <[email protected]> >> >>> To: rsyslog-users <[email protected]> >> >>> Subject: Re: [rsyslog] cee json + mmsequence >> >>> >> >>> >> >>> mmsequence is deprecated? So which plugin instead? >> >>> >> >>> 2015-05-22 21:15 GMT+08:00 singh.janmejay <[email protected]>: >> >>> >> >>>> Cool, thats the answer I was looking for. >> >>>> >> >>>> >> >>>> On Fri, May 22, 2015 at 5:55 PM, Rainer Gerhards >> >>>> <[email protected]> wrote: >> >>>>> >> >>>>> 2015-05-21 17:40 GMT+02:00 singh.janmejay >> >>>>> <[email protected]>: >> >>>>>> >> >>>>>> Are $/ variable manipulations supposed to be atomic? >> >>>>> >> >>>>> >> >>>>> It depends on what you mean with "atomic". The value returned should >> >>>>> be consistent, that for sure. But if you use >> >>>>> >> >>>>> $/var = $/var + 1 >> >>>> >> >>>> >> >>>> Yes, I expect it to jump, but not get corrupted. >> >>>> >> >>>>> >> >>>>> and $/var is 16 when this starts, the result can very well be 18 if >> >>>>> another thread executed the exact same sequence. I think for this >> >>>>> use >> >>>>> there was an atomic increment function planned. I guess I never >> >>>>> implemented it, though. >> >>>>> >> >>>>>> >> >>>>>> Eg. $/var % 16 should never produce any number greater than 15, but >> >>>>>> it >> >>>> >> >>>> does. >> >>>> >> >>>> This will not be a problem if we use atomic read. >> >>>> Since you have made it clear that the idea was to make it >> >>>> atomic(which >> >>>> means this is a bug), I'll fix this. >> >>>> >> >>>>> >> >>>>> as David said, that's a bug. >> >>>>> >> >>>>>> >> >>>>>> Are they non-atomic by design? If so, is mmsequence the right thing >> >>>>>> to >> >>>>>> use (assuming it has atomic counters, I haven't checked if it >> >>>>>> really >> >>>>>> does)? >> >>>>> >> >>>>> >> >>>>> mmsequence is deprecated. >> >>>>> >> >>>>>> >> >>>>>> There are few options on the table, I guess. >> >>>>> >> >>>>> >> >>>>> before I take a shot at these, let's see if we had the same >> >>>>> understanding of "atomic". >> >>>> >> >>>> >> >>>> We do. >> >>>> >> >>>>> >> >>>>> Rainer >> >>>>>> >> >>>>>> 1. If $/ variables are supposed to be atomic, we just fix the impl. >> >>>>>> 2. We use mmsequence, and keep it alive (assuming it has atomic >> >>>> >> >>>> counters) >> >>>>>> >> >>>>>> 3. We fix mmsequence to implement atomic counters. >> >>>>>> 4. If $/ is not supposed to be atomic (by design), we implement $@ >> >>>>>> which is thread-local, so people can get atomic counters within >> >>>>>> thread >> >>>>>> (evenness in data(for load-balancing or sampling kind of usecase) >> >>>>>> is >> >>>>>> maintained by the fact that a single thread will pump a lot of >> >>>>>> messages, so across all threads it'll approximately be as even as >> >>>>>> it >> >>>>>> is in a single-thread. >> >>>>>> 5. We implement a random-number construct (which i believe either >> >>>>>> mmsequence or mmcount had) which allows acceptable load-balancing >> >>>>>> or >> >>>>>> sampling etc. >> >>>>>> >> >>>>>> Thoughts? >> >>>>>> >> >>>>>> On Wed, Dec 3, 2014 at 11:12 PM, David Lang <[email protected]> wrote: >> >>>>>>> >> >>>>>>> On Wed, 3 Dec 2014, Brian Knox wrote: >> >>>>>>> >> >>>>>>>> So two questions then: >> >>>>>>>> >> >>>>>>>> 1. In the case of using mmjsonparse on messages where we do not >> >>>>>>>> know >> >>>>>>>> ahead >> >>>>>>>> of time what all the. json attributes will be, is there a var for >> >>>>>>>> a >> >>>> >> >>>> place >> >>>>>>>> >> >>>>>>>> in the tree where we can get all attributes that were in the cee >> >>>> >> >>>> message? >> >>>>>>> >> >>>>>>> >> >>>>>>> >> >>>>>>> There are three 'classes' of variables >> >>>>>>> >> >>>>>>> $! are 'normal' variables, the place that mmjsonparse, etc put the >> >>>> >> >>>> results >> >>>>>>> >> >>>>>>> of parsing the cee message by default. >> >>>>>>> >> >>>>>>> $. are 'local' variables, they are a place you can put stuff that >> >>>>>>> you >> >>>> >> >>>> don't >> >>>>>>> >> >>>>>>> want to end up in a message when you output $! >> >>>>>>> >> >>>>>>> $\ are 'global' variables, unlike the others, they persist from >> >>>> >> >>>> message to >> >>>>>>> >> >>>>>>> message. >> >>>>>>> >> >>>>>>> All of these variables are a tree, where you can have $!foo!bar >> >>>>>>> and >> >>>>>>> $!foo!baz. If you just reference $!foo it will output both bar and >> >>>>>>> baz >> >>>> >> >>>> as >> >>>>>>> >> >>>>>>> JSON >> >>>>>>> >> >>>>>>> so to see everything, parse the message and then output $! >> >>>>>>> >> >>>>>>>> 2. If mmsequence is being deprecated, is there a pointer to how >> >>>>>>>> to >> >>>>>>>> increment global variables to provide a sequence? >> >>>>>>> >> >>>>>>> >> >>>>>>> >> >>>>>>> set $\var = $\var + 1; >> >>>>>>> >> >>>>>>>> 3. This morning I remembered the "mmcount" module - is this also >> >>>>>>>> being >> >>>>>>>> deprecated in favor of global variables? >> >>>>>>> >> >>>>>>> >> >>>>>>> >> >>>>>>> probably. >> >>>>>>> >> >>>>>>> David Lang >> >>>>>>> >> >>>>>>> >> >>>>>>>> Brian >> >>>>>>>> >> >>>>>>>> On Wed, Dec 3, 2014 at 3:52 AM, Rainer Gerhards < >> >>>> >> >>>> [email protected]> >> >>>>>>>> >> >>>>>>>> wrote: >> >>>>>>>> >> >>>>>>>>> 2014-12-02 21:59 GMT+01:00 Brian Knox <[email protected]>: >> >>>>>>>>> >> >>>>>>>>>> We have a case where we are receiving messages in @cee: format, >> >>>>>>>>>> and >> >>>>>>>>>> would >> >>>>>>>>>> like to add a sequence variable generated via mmsequence to the >> >>>> >> >>>> message. >> >>>>>>>>>> >> >>>>>>>>>> >> >>>>>>>>>> Before I spend too much time digging into this I wanted to ask >> >>>>>>>>>> if, >> >>>> >> >>>> using >> >>>>>>>>>> >> >>>>>>>>>> some template hackery and json subtrees, this is something that >> >>>> >> >>>> sounds >> >>>>>>>>>> >> >>>>>>>>>> possible - and if maybe someone else was already doing it. >> >>>>>>>>>> >> >>>>>>>>>> >> >>>>>>>>> David has answered the real question, just let me add that >> >>>> >> >>>> mmsequence was >> >>>>>>>>> >> >>>>>>>>> a >> >>>>>>>>> work-around for v7 where global variables were not available. It >> >>>>>>>>> is >> >>>> >> >>>> now >> >>>>>>>>> >> >>>>>>>>> deprecated and will probably removed in a couple of month. >> >>>>>>>>> >> >>>>>>>>> I'll check the doc if it tells this clearly enough. >> >>>>>>>>> >> >>>>>>>>> Rainer >> >>>>>>>>> >> >>>>>>>>>> Thanks! >> >>>>>>>>>> Brian >> >>>>>>>>>> _______________________________________________ >> >>>>>>>>>> rsyslog mailing list >> >>>>>>>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog >> >>>>>>>>>> http://www.rsyslog.com/professional-services/ >> >>>>>>>>>> What's up with rsyslog? Follow https://twitter.com/rgerhards >> >>>>>>>>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by >> >>>>>>>>>> a >> >>>> >> >>>> myriad >> >>>>>>>>>> >> >>>>>>>>>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST >> >>>>>>>>>> if >> >>>> >> >>>> you >> >>>>>>>>>> >> >>>>>>>>>> DON'T LIKE THAT. >> >>>>>>>>>> >> >>>>>>>>> _______________________________________________ >> >>>>>>>>> rsyslog mailing list >> >>>>>>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog >> >>>>>>>>> http://www.rsyslog.com/professional-services/ >> >>>>>>>>> What's up with rsyslog? Follow https://twitter.com/rgerhards >> >>>>>>>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by >> >>>>>>>>> a >> >>>> >> >>>> myriad >> >>>>>>>>> >> >>>>>>>>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST >> >>>>>>>>> if >> >>>> >> >>>> you >> >>>>>>>>> >> >>>>>>>>> DON'T LIKE THAT. >> >>>>>>>>> >> >>>>>>>> _______________________________________________ >> >>>>>>>> rsyslog mailing list >> >>>>>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog >> >>>>>>>> http://www.rsyslog.com/professional-services/ >> >>>>>>>> What's up with rsyslog? Follow https://twitter.com/rgerhards >> >>>>>>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a >> >>>> >> >>>> myriad >> >>>>>>>> >> >>>>>>>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST >> >>>>>>>> if >> >>>> >> >>>> you DON'T >> >>>>>>>> >> >>>>>>>> LIKE THAT. >> >>>>>>>> >> >>>>>>> _______________________________________________ >> >>>>>>> rsyslog mailing list >> >>>>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog >> >>>>>>> http://www.rsyslog.com/professional-services/ >> >>>>>>> What's up with rsyslog? Follow https://twitter.com/rgerhards >> >>>>>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a >> >>>> >> >>>> myriad of >> >>>>>>> >> >>>>>>> sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if >> >>>>>>> you >> >>>> >> >>>> DON'T >> >>>>>>> >> >>>>>>> LIKE THAT. >> >>>>>> >> >>>>>> >> >>>>>> >> >>>>>> >> >>>>>> -- >> >>>>>> Regards, >> >>>>>> Janmejay >> >>>>>> http://codehunk.wordpress.com >> >>>>>> _______________________________________________ >> >>>>>> rsyslog mailing list >> >>>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog >> >>>>>> http://www.rsyslog.com/professional-services/ >> >>>>>> What's up with rsyslog? Follow https://twitter.com/rgerhards >> >>>>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a >> >>>> >> >>>> myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT >> >>>> POST if >> >>>> you DON'T LIKE THAT. >> >>>>> >> >>>>> _______________________________________________ >> >>>>> rsyslog mailing list >> >>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog >> >>>>> http://www.rsyslog.com/professional-services/ >> >>>>> What's up with rsyslog? Follow https://twitter.com/rgerhards >> >>>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a >> >>>>> myriad >> >>>> >> >>>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if >> >>>> you >> >>>> DON'T LIKE THAT. >> >>>> >> >>>> >> >>>> >> >>>> -- >> >>>> Regards, >> >>>> Janmejay >> >>>> http://codehunk.wordpress.com >> >>>> _______________________________________________ >> >>>> rsyslog mailing list >> >>>> http://lists.adiscon.net/mailman/listinfo/rsyslog >> >>>> http://www.rsyslog.com/professional-services/ >> >>>> What's up with rsyslog? Follow https://twitter.com/rgerhards >> >>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a >> >>>> myriad >> >>>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if >> >>>> you >> >>>> DON'T LIKE THAT. >> >>>> >> >>> _______________________________________________ >> >>> rsyslog mailing list >> >>> http://lists.adiscon.net/mailman/listinfo/rsyslog >> >>> http://www.rsyslog.com/professional-services/ >> >>> What's up with rsyslog? Follow https://twitter.com/rgerhards >> >>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a >> >>> myriad >> >>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you >> >>> DON'T >> >>> LIKE THAT. >> >>> >> >> _______________________________________________ >> >> rsyslog mailing list >> >> http://lists.adiscon.net/mailman/listinfo/rsyslog >> >> http://www.rsyslog.com/professional-services/ >> >> What's up with rsyslog? Follow https://twitter.com/rgerhards >> >> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a >> >> myriad of >> >> sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you >> >> DON'T >> >> LIKE THAT. >> > >> > >> > >> > -- >> > Regards, >> > Janmejay >> > http://codehunk.wordpress.com >> > _______________________________________________ >> > rsyslog mailing list >> > http://lists.adiscon.net/mailman/listinfo/rsyslog >> > http://www.rsyslog.com/professional-services/ >> > What's up with rsyslog? Follow https://twitter.com/rgerhards >> > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad >> > of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you >> > DON'T >> > LIKE THAT. >> _______________________________________________ >> rsyslog mailing list >> http://lists.adiscon.net/mailman/listinfo/rsyslog >> http://www.rsyslog.com/professional-services/ >> What's up with rsyslog? Follow https://twitter.com/rgerhards >> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad >> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T >> LIKE THAT. -- Regards, Janmejay http://codehunk.wordpress.com _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
