So some of this hits exactly on what I've been working on lately. I was thinking about discussing it at the hackathon.
We were looking at having some kind of simple, distributed key/value storage system for storing chunks of stuff (html, xml, whatever) with simple, namespaced keys. Stuff that doesn't belong in Mysql, and fits more along the lines of Berkeley DB, but is accessible over the network and can be accessed with a protocol instead of an API. We already use memcached, and we use at least 4 different languages, so coming up with a new client library for each of those didn't sound so fun, so the memcached protocol fit almost perfectly. There were a few other things we wanted: persistent storage, some redundancy, deterministic key routing, and ease of configuration. I looked at two options to accomplish these things. The first was to extend the memcached source to handle it all, and the other was as an Apache 2 module, which is the route I ended up taking. Some things that made the Apache route sound pretty good: first, the ability to layer an HTTP interface over the storage access was already there, although the initial version is actually a protocol handler that doesn't do HTTP (yet). My idea was that eventually, some kind of simple REST interface could map directly to the same functions that also handled the memcached protocol calls. The second was the existence of an easy way to configure all of these things. And the third was modularity. Although my current version is a hardcoded memcached-in/memcached-out proxy, my plan is to wire up something similar to mod_cache whereby the protocol itself is handled in one module, and one or more storage providers can be written to handle what goes in/comes out of the backend, so that storage can be just about anything, as long as you implement the callback interface to fit within the "SET/GET/DELETE" etc. interface. So yeah, I've been thinking about/poking at some of this stuff for a little while now, and I'd love to talk more about the details. Josh (snipped out the quoted stuff here) ....
