I realize that I may be slightly pushing the limits. I have been looking at membase/couchdb/couchbase however the persistence options are so-so: I don't want to lose *any* data so the idea of "it will be written to the disk later" is not really satisfying.
Unfortunately, like I said, I am aiming at highly volatile data, so I expect a read/write ratio of about 3/1; most components I have seen seem built in the expectation of a much more highly skewed ratio (say 100/1) and favor low-latency over persistence which does not suit me. Anyway, I have a couple leads already [1], I am just aiming at improving performance. [1] I have a couple options open to me, although only one does not require fiddling with memcached code: - Option 1 consists in versioning the list of memcached instances in the cluster and introduce a "pause" in the system when a change to the list is done so that all clients have the time to realize it's changing and that some pieces of data will be relocated. It can work with the current memcached version and I have control over all the clients so can impose a wrapper around the memcached client api. - Option 2 consists in finding a memcached version with a getl/unl command (which I read about in the couchbase documentation), it seems a natural extension to memcached. I would be acceptable to get just the patch and patch the actual version to get just that. - Option 3 consists in implementing a rename command within memcached. Least preferable obviously since I would have to come up with the code and I am blissfully ignorant of the internals of memcached at the moment. On other hand it could be a worthy contribution I think since this is not something that can be emulated. On Saturday, June 16, 2012 7:47:58 PM UTC+2, Nelz wrote: > > The problem that you are having is that you aren't using memcached as > a cache in that case... You are using it as a data-store. > > I'd suggest using other tools (Membase? Redis?) that fit your > requirements, because if you keep trying to use memcached in a > use-case for other than it is designed for, you will continually run > into problems. > > - Nelz > > On Sat, Jun 16, 2012 at 10:17 AM, Matthieu M. > <[email protected]> wrote: > > > > > > On Saturday, June 16, 2012 12:08:31 AM UTC+2, Henrik Schröder wrote: > >> > >> Why don't you just take the cache misses and recreate your data from > the > >> underlying datastore? If you are using consistent hashing and you > change > >> your cluster slightly, you lose only a fraction of your cached values. > >> > >> > >> /Henrik > >> > >> On Fri, Jun 15, 2012 at 7:05 PM, Matthieu M. < > [email protected]> > >> wrote: > >>> > >>> I did not find a "rename" operation as I combed the docs, which would > >>> have been my savior here, so I would like to know how everyone else > handles > >>> this situation. > >>> > > > > Unfortunately I am counting on using memcached for highly volatile data > with > > asynchronous reliable storage (I am thinking thousands to tens of > thousands > > of writes per seconds on those simple counters), so in case of cache > miss I > > have to wait for a while (tens of seconds to minutes) before I can > reliably > > know that all updates were flushed to the data store and restore. > > > > I just came over an extension to the memcached protocol today (get with > > lock: getl and its counterpart unlock: unl) that is experimented with > and > > could help achieve what I want, unfortunately it might be a while before > it > > is available in memcached core. > > > > -- Matthieu >
