On Fri, May 3, 2013 at 11:31 AM, David Lang <[email protected]> wrote: > On Fri, 3 May 2013, Rainer Gerhards wrote: > > On Fri, May 3, 2013 at 11:15 AM, David Lang <[email protected]> wrote: >> >> On Fri, 3 May 2013, Rainer Gerhards wrote: >>> >>> tracking state between messages will require a major set of changes to >>> >>>> rsyslog internals (think of of problems like having multiple threads >>>>> processing messages and the need for them to start sharing state >>>>> information (potentially a LOT of state information) >>>>> >>>>> Let me jump on that one (waited for the discussion). I think global >>>>> >>>> variables are not that hard, I just did not do them as I questioned if >>>> there is any use case. In essence it just needs to have some separate >>>> namespace. >>>> >>>> Do you see use cases where this would make sense? >>>> >>>> >>> Setting variables that get used for other messages sounds useful, I can >>> think of several ways to use this (a simple one would be to tag if >>> messages >>> happen during business hours or not, triggered by special log messages. >>> >>> My main concern is around the locking needed to update such global >>> variables when there are lots of threads (and lots of global variables) >>> around. Since we are talking about variable length strings there is no >>> way >>> to do atomic updates, so you will need to have reader and writer locks or >>> some form of rcu to manage the updates. Just the overhead of grabbing and >>> releasing the read lock for each message is enough overhead to be worried >>> about, even when no updates happen. >>> >>> Now, as always, I may be missing some trick that eliminates this worry, >>> if >>> so I'd love to hear about it. >>> >> >> >> I haven't yet thought hard about it (and not about to do due to time >> constraints), but I think there is no way around read/write looks. As long >> as there are few writers, though, the effect should be limited. However, >> if >> global vars are very frequently used, you'll notice... >> > > Actually, I'm concerned about the overhead of just grabbing and releasing > the reader lock. This consists of doing a atomic incrament/decrament of > some memory location, doing this across a bunch of threads at a high rate > is expensive (remember we can do 1m+ logs/sec, that's a LOT of inc/dec > atomic ops. I expect this to be expensive enough to make a noticeable dent > in the log processing rate. > > some form of rcu, where the readers don't need to lock anything and the > writers can prepare a new variable table and do an atomic write of the > table location would have some chance of working well enough. But even with > this, var=var+1 will be a problem, because the writer will not always know > the latest value of var (rcu allows for a thread to continue to operate > with an old version of the variable if it's updated after the thread starts > processing a message, so you could have tao threads see that var=3 and try > to add to it, the result will be var=4 instead of var=5 like it should be. > However, for non-math type operations, this would work (acknowleging the > race conditions if two threads try to set different results)
I agreeit has subtle issues, but unfortuantely I cannot dig deeper right now (too much on the table, especially given the fact that I took out a couple of days to get rid of Exchange... finally! ;)). Rainer _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
