On Fri, 3 May 2013, Rainer Gerhards wrote:
On Fri, May 3, 2013 at 11:15 AM, David Lang <[email protected]> wrote:
On Fri, 3 May 2013, Rainer Gerhards wrote:
tracking state between messages will require a major set of changes to
rsyslog internals (think of of problems like having multiple threads
processing messages and the need for them to start sharing state
information (potentially a LOT of state information)
Let me jump on that one (waited for the discussion). I think global
variables are not that hard, I just did not do them as I questioned if
there is any use case. In essence it just needs to have some separate
namespace.
Do you see use cases where this would make sense?
Setting variables that get used for other messages sounds useful, I can
think of several ways to use this (a simple one would be to tag if messages
happen during business hours or not, triggered by special log messages.
My main concern is around the locking needed to update such global
variables when there are lots of threads (and lots of global variables)
around. Since we are talking about variable length strings there is no way
to do atomic updates, so you will need to have reader and writer locks or
some form of rcu to manage the updates. Just the overhead of grabbing and
releasing the read lock for each message is enough overhead to be worried
about, even when no updates happen.
Now, as always, I may be missing some trick that eliminates this worry, if
so I'd love to hear about it.
I haven't yet thought hard about it (and not about to do due to time
constraints), but I think there is no way around read/write looks. As long
as there are few writers, though, the effect should be limited. However, if
global vars are very frequently used, you'll notice...
Actually, I'm concerned about the overhead of just grabbing and releasing the
reader lock. This consists of doing a atomic incrament/decrament of some memory
location, doing this across a bunch of threads at a high rate is expensive
(remember we can do 1m+ logs/sec, that's a LOT of inc/dec atomic ops. I expect
this to be expensive enough to make a noticeable dent in the log processing
rate.
some form of rcu, where the readers don't need to lock anything and the writers
can prepare a new variable table and do an atomic write of the table location
would have some chance of working well enough. But even with this, var=var+1
will be a problem, because the writer will not always know the latest value of
var (rcu allows for a thread to continue to operate with an old version of the
variable if it's updated after the thread starts processing a message, so you
could have tao threads see that var=3 and try to add to it, the result will be
var=4 instead of var=5 like it should be. However, for non-math type operations,
this would work (acknowleging the race conditions if two threads try to set
different results)
David Lang
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
