I'm exploring it. I'm accessing memcached from sec now for external
data and it sure seems fast.
Thanks,
-Joe
On Fri, Sep 23, 2011 at 8:27 PM, John P. Rouillard <rou...@cs.umb.edu> wrote:
>
> Hi Joe:
>
> In message
> <CA+=j5hv1ydmjipteyju9ppbimufpayaq0wdjmomm17i46iv...@mail.gmail.com> ,
> Joe Prosser writes:
>>Has anyone thought of or built an SEC installation with the contexts
>>saved on a memcached server?
>>This strikes me as a possibly good way to address scalability
>>challenges if you can get past lack of atomicity.
>
> By scalability I assume you mean running multiple SEC proceses on one
> or more hosts (to use multiple cpu/cores/memory) and have them all
> share the same pool of contexts.
>
> One thing that concerns me is that memcached is a cache tool and not a
> permanent storage tool IIRC it can evict things from cache (which I
> think would lose a context).
>
> IIUC each memcached operation is atomic, but if you want to update a
> context you have to do a get/set which isn't atomic (the value of the
> context may change after the get but before the set and you would lose
> data). I guess that's what you mean by "lack of atomicity".
>
> You could of lock a context using the add method which does a set only
> if it is currently not present. If it fails you know somebody else
> holds the lock. With add you can set an expire/timeout value so the
> lock will go away at some time in case process with pid $mypid dies.
> This should allow an atomic update/delete of a context using:
>
> add($contextname ".LOCK", "$mypid", $sometimeoutvalue)
> $context{$context_name} = ..., $context{$context_name} ..., ...
> delete($contextname ".LOCK")
>
> Cache::Memcached::Tie looks like it may fit the bill to minimize the
> number of changes with the exception that dumping the state of sec I
> think iterates over the hash and first/next isn't implemented
> IIUC. You could keep a list of all contexts in memcached and use that
> in your iterator if you needed to.
>
> It would require adding in the lock calls but I think Risto wrote the
> code modularly enough that there are a handful of functions where
> locking would need to be added. However dealing with the possible loss
> of context data is the big red flag.
>
> It would be an interesting POC to try it and see. The question is how
> much slower would it be and at what point would adding sec processes
> result in diminishing returns. But it would be a good way to see if
> storing contexts in something distributed could be done reasonably.
>
> --
> -- rouilj
> John Rouillard
> ===========================================================================
> My employers don't acknowledge my existence much less my opinions.
>
>
------------------------------------------------------------------------------
All of the data generated in your IT infrastructure is seriously valuable.
Why? It contains a definitive record of application performance, security
threats, fraudulent activity, and more. Splunk takes this data and makes
sense of it. IT sense. And common sense.
http://p.sf.net/sfu/splunk-d2dcopy2
_______________________________________________
Simple-evcorr-users mailing list
Simple-evcorr-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/simple-evcorr-users
