Brandon: http://code.google.com/p/googleappengine/issues/detail?id=6078
On Nov 25, 3:20 am, "Brandon Wirtz" <[email protected]> wrote: > Nick can you confirm my suspicion that MemCache does not scale in size with > the number of instances? > > The bigger ask here was really the ability to specify a size of memcache. > The preferred Muli-tenant model breaks down if a shared resource doesn't > scale with the "load" . If I have 1 gig of data being accessed daily > memcache helps a lot. When I push 6 gigs a day my cache hit ratio falls to > nothing. So right now I'm far better with lots of apps optimized for > 500-ish megs of unique data than I am one at 60gigs. That feels backwards > of what I want in a cloud environment. > > If there my assumption is wrong I need to go looking at why this is > happening. > > From: [email protected] > [mailto:[email protected]] On Behalf Of Nick Johnson > Sent: Thursday, November 24, 2011 5:52 PM > To: [email protected] > Subject: Re: [google-appengine] Potential Feature Request Memcache > Allocation and Delayed Writes > > Hi Brandon, > > On Thu, Nov 24, 2011 at 8:05 PM, Brandon Wirtz <[email protected]> wrote: > > Realizing my app is different than from most everybody elses.. > > I got to thinking about the thread where we were talking about reading keys > from memcache. I know all the reasons this is a bad idea. But I got to > thinking what if it wasn't? > > My app is just a big optimized cache, but I rely on the 3 tiers of storage > to make it all work and do so quickly. The sum of all the data for the day > is about 2 gigs. In a virtual machine environment I would typically allocate > a bunch of ram and every so often dump that to longer term storage, but > since most my caching is measured in minutes, some is in days, and the > longest I ever care about data is a month. the only reason I need long term > storage is so that when the memory gets reset, or a new "instance" comes > online that I don't have a 100% miss rate. > > The system you describe won't evict data unless you explicitly choose to do > so - you are in full control of cache eviction. This is not the case for > memcache, where memcache may choose to evict items without notice. I don't > think your proposal really applies to memcache, since by the time you do > your 'dump', some indeterminate amount of your original data will have > already been evicted from the cache. > > A much better approach if you don't mind data loss is a write-behind cache, > where you schedule a task to write the data to the datastore after each time > you update it in memcache. > > -Nick Johnson > > Why can't I do that with Memcache? Allocate 2 gigs, populate it with data > only on a version change. Once a day take all the values and dump them back > to datastore so that if the world ends that I don't have to start from > nothing. (maybe only write all the values that have an expiration so many > hours away) > > Since Backends share Memcache this "long" operation could be a scheduled > task and execute in the background. > > In my case this would save a lot of cycles since my writes are Local Memory, > MemCache, Datastore. And I do so with every piece of data because I can't > count on getting a hit from Memory or MemCache because of their volatility. > > But if I had a set amount of Memcache I wouldn't need to worry, it wouldn't > be volatile, and Google Could charge me for the resource. Doesn't even have > to be perfectly non-volatile because even if I only "back-up" 75% of the > data that's fine it is just a cache. > > -- > You received this message because you are subscribed to the Google Groups > "Google App Engine" group. > To post to this group, send email to [email protected]. > To unsubscribe from this group, send email to > [email protected] > <mailto:google-appengine%[email protected]> . > For more options, visit this group > athttp://groups.google.com/group/google-appengine?hl=en. > > -- > Nick Johnson, Developer Programs Engineer, App Engine > > -- > You received this message because you are subscribed to the Google Groups > "Google App Engine" group. > To post to this group, send email to [email protected]. > To unsubscribe from this group, send email to > [email protected]. > For more options, visit this group > athttp://groups.google.com/group/google-appengine?hl=en. -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to [email protected]. To unsubscribe from this group, send email to [email protected]. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
