If you don't want to lose any data, how will you handle the case when memcached evicts some of your data? How will you handle the case when a memcached server crashes? Those two events are more likely to happen than your planned cluster changes.
/Henrik On Sun, Jun 17, 2012 at 4:08 PM, Matthieu M. <[email protected]>wrote: > I realize that I may be slightly pushing the limits. > > I have been looking at membase/couchdb/couchbase however the persistence > options are so-so: I don't want to lose *any* data so the idea of "it will > be written to the disk later" is not really satisfying. > > Unfortunately, like I said, I am aiming at highly volatile data, so I > expect a read/write ratio of about 3/1; most components I have seen seem > built in the expectation of a much more highly skewed ratio (say 100/1) and > favor low-latency over persistence which does not suit me. > > Anyway, I have a couple leads already [1], I am just aiming at improving > performance. > > > [1] I have a couple options open to me, although only one does not require > fiddling with memcached code: > > - Option 1 consists in versioning the list of memcached instances in the > cluster and introduce a "pause" in the system when a change to the list is > done so that all clients have the time to realize it's changing and that > some pieces of data will be relocated. It can work with the current > memcached version and I have control over all the clients so can impose a > wrapper around the memcached client api. > - Option 2 consists in finding a memcached version with a getl/unl command > (which I read about in the couchbase documentation), it seems a natural > extension to memcached. I would be acceptable to get just the patch and > patch the actual version to get just that. > - Option 3 consists in implementing a rename command within memcached. > Least preferable obviously since I would have to come up with the code and > I am blissfully ignorant of the internals of memcached at the moment. On > other hand it could be a worthy contribution I think since this is not > something that can be emulated. > > > > On Saturday, June 16, 2012 7:47:58 PM UTC+2, Nelz wrote: >> >> The problem that you are having is that you aren't using memcached as >> a cache in that case... You are using it as a data-store. >> >> I'd suggest using other tools (Membase? Redis?) that fit your >> requirements, because if you keep trying to use memcached in a >> use-case for other than it is designed for, you will continually run >> into problems. >> >> - Nelz >> >> On Sat, Jun 16, 2012 at 10:17 AM, Matthieu M. >> <[email protected]> wrote: >> > >> > >> > On Saturday, June 16, 2012 12:08:31 AM UTC+2, Henrik Schröder wrote: >> >> >> >> Why don't you just take the cache misses and recreate your data from >> the >> >> underlying datastore? If you are using consistent hashing and you >> change >> >> your cluster slightly, you lose only a fraction of your cached values. >> >> >> >> >> >> /Henrik >> >> >> >> On Fri, Jun 15, 2012 at 7:05 PM, Matthieu M. < >> [email protected]> >> >> wrote: >> >>> >> >>> I did not find a "rename" operation as I combed the docs, which would >> >>> have been my savior here, so I would like to know how everyone else >> handles >> >>> this situation. >> >>> >> > >> > Unfortunately I am counting on using memcached for highly volatile data >> with >> > asynchronous reliable storage (I am thinking thousands to tens of >> thousands >> > of writes per seconds on those simple counters), so in case of cache >> miss I >> > have to wait for a while (tens of seconds to minutes) before I can >> reliably >> > know that all updates were flushed to the data store and restore. >> > >> > I just came over an extension to the memcached protocol today (get with >> > lock: getl and its counterpart unlock: unl) that is experimented with >> and >> > could help achieve what I want, unfortunately it might be a while >> before it >> > is available in memcached core. >> > >> > -- Matthieu >> >
