Hi Joe:
In message
<CA+=j5hv1ydmjipteyju9ppbimufpayaq0wdjmomm17i46iv...@mail.gmail.com> ,
Joe Prosser writes:
>Has anyone thought of or built an SEC installation with the contexts
>saved on a memcached server?
>This strikes me as a possibly good way to address scalability
>challenges if you can get past lack of atomicity.
By scalability I assume you mean running multiple SEC proceses on one
or more hosts (to use multiple cpu/cores/memory) and have them all
share the same pool of contexts.
One thing that concerns me is that memcached is a cache tool and not a
permanent storage tool IIRC it can evict things from cache (which I
think would lose a context).
IIUC each memcached operation is atomic, but if you want to update a
context you have to do a get/set which isn't atomic (the value of the
context may change after the get but before the set and you would lose
data). I guess that's what you mean by "lack of atomicity".
You could of lock a context using the add method which does a set only
if it is currently not present. If it fails you know somebody else
holds the lock. With add you can set an expire/timeout value so the
lock will go away at some time in case process with pid $mypid dies.
This should allow an atomic update/delete of a context using:
add($contextname ".LOCK", "$mypid", $sometimeoutvalue)
$context{$context_name} = ..., $context{$context_name} ..., ...
delete($contextname ".LOCK")
Cache::Memcached::Tie looks like it may fit the bill to minimize the
number of changes with the exception that dumping the state of sec I
think iterates over the hash and first/next isn't implemented
IIUC. You could keep a list of all contexts in memcached and use that
in your iterator if you needed to.
It would require adding in the lock calls but I think Risto wrote the
code modularly enough that there are a handful of functions where
locking would need to be added. However dealing with the possible loss
of context data is the big red flag.
It would be an interesting POC to try it and see. The question is how
much slower would it be and at what point would adding sec processes
result in diminishing returns. But it would be a good way to see if
storing contexts in something distributed could be done reasonably.
--
-- rouilj
John Rouillard
===========================================================================
My employers don't acknowledge my existence much less my opinions.
------------------------------------------------------------------------------
All of the data generated in your IT infrastructure is seriously valuable.
Why? It contains a definitive record of application performance, security
threats, fraudulent activity, and more. Splunk takes this data and makes
sense of it. IT sense. And common sense.
http://p.sf.net/sfu/splunk-d2dcopy2
_______________________________________________
Simple-evcorr-users mailing list
Simple-evcorr-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/simple-evcorr-users
