Anyone interested in the patches I did for the CAS implementation for
memcache 1.2.3, and libmemcache? It's prototype and could be improved
but its working :-)
Chris Goffinet
[EMAIL PROTECTED]
On Sep 8, 2007, at 6:44 PM, dormando wrote:
Thanks for the example, that cleared it right up! Personally I'll
agree that this might be the first actually useful server extension
I've heard of for a long while. It removes (at least) one round
trip, reduces polling, and helps deal with the LRU for the usual
workaround case.
Are there enough potential server-side operations to really justify
a scripting interface? When it's been brought up in meetings, the
response has varied from interest, to a warning about crossing
streams or something.
Both a CAS and a scripting interface provide a foundation for
building atomic data structure manipulation without having to
define all of the data structures and all of the operations at the
protocol level. Each has downside. With CAS, you may have to
retry a few times under contention and this would be amplified by
the latency. With scripting, you may easily slow down the whole
server, and you have to deal with how parameters are passed in and
results are returned -- and this would mean very different things
by protocol.
I remember floating the idea of embedding lua at the last meet and
not especially getting torn up by it. Paul Querna had some good
ideas that I've been wanting to get into my other C/lua project
sometime soon. It's in my big fancy TODO to attempt a proof of
concept for lua + memcached after I get a few more releases of my
other lua project out the door (so a month or two out).
Embedding lua should be trivial to try out, and has some nifty ideas
to it:
- Dynamically add/remove commands to the server (Paul's idea?).
- Any wild n' crazy "needs to be atomic" operation you can think of.
- Easily implement a slab-realloc thread, using pluggable strategies.
- Coordination algorithms, subscription connections, blah blah.
Tons of downsides:
- Gives new users a swiss army chainsaw with which to decapitate
themselves.
- High end users are unlikely to use it at all, given their needs
for the binary protocol already. Needing a sun T2000 or quad socket
barcelona to run memcached isn't what was advertised.
- Will add a lot more latency to any command which uses scripting,
even something as fast as lua, with pre-compiled commands. You could
try a scripting approach like Varnish and dynamically compile/link
in plugins?...
- If using lua, the garbage collector will need to be carefully
managed. I'm going to be tuning this a lot for my own project, which
needs to manage potentially hundreds of millions of objects over
time. Otherwise memcached will randomly block operations.
- Again, if using lua at least, the default allocator will need to
be replaced with memcached's own slabber, or a new slabber.
Otherwise the same malloc() issues which we've been avoiding until
now will come back in spades.
- Turns threading into a nightmare. There're a few options with lua,
but you end up having to do tons of internal locking. If you decide
to run completely independent lua instances per thread instead,
you're back to square one without anything being atomic.
- Again with the swiss army chainsaw.
Any better/different alternatives? Maybe don't even try scripting? :)
I'd be up for attempting a proof-of-concept lua embedding once the
binary protocol stuff settles a little more. Could be accelerated if
someone who sucks less at C helps out or we'd meet up somewhere and
just write it.
*shrug*,
-Dormando
