Memcache doesn't work for this application. Completely aside from reliability issues, there's no memcache instruction for "give me all the data". At best you can fetch some number of individual keys, but that brings up two problems:
1) Trying to do a batch fetch of 10k keys (or more) 2) How do you know what keys to fetch in the first place? #2 is intractable because any practical solution would essentially eliminate the need for memcache in the first place. The problem is pretty easily stated: Collect 10k score submissions in 5s and be able to provide a sorted leaderboard 5s later. GAE does not offer any practical facility capable of accomplishing this. Jeff On Fri, Aug 10, 2012 at 1:45 PM, Kristopher Giesing <[email protected]> wrote: > Good point. I had unconsciously assumed that a replicated memcached > solution would be much more efficient than a replicated data store solution, > but if the replication and not the disk I/O is what's expensive, then that > assumption falls down. > > Generally speaking, I'm having difficulty architecting my own app in such a > way that it's robust across instances that may be "very distant" from one > another. Right now I'm relying on optimistic concurrency in the datastore, > but it means my app is much slower than, say, a single instance that held > everything in RAM. It's been tempting to just drop the datastore entirely > and limit myself to a single instance, but GAE doesn't seem designed to > scale down that way; front end instances spin up apparently for no reason, > and back end instances, as this thread reveals, seem to have more > performance limitations than one would expect, so I have no idea whether a > single instance would actually be able to service my user base. > > - Kris > > On Friday, August 10, 2012 2:30:35 AM UTC-7, Joakim wrote: >> >> By *guaranteed*, do you mean that this should be kept synchronized across >> multiple data centers? I ask because in the case of a data center emergency, >> your app spins up at a different data center. The HR datastore is always in >> sync, the memcache is always local to the DC. >> Comparing the put-latency of the datastore and memcache for the last few >> days, the datastore generally keeps under 50 ms while memcache tends to stay >> below 30 ms. I'd say the datastore isn't so bad, and adding sync to memcache >> would probably bring that number up a bit. At that point, this new feature >> is starting to look a bit like the current HR datastore. Add properties for >> insertion date/time etc, and it could be a queue, right? >> >> I don't mean to shoot you down, but to make you specify what it really is >> you want, and how it relates to the current offering. >> > -- > You received this message because you are subscribed to the Google Groups > "Google App Engine" group. > To view this discussion on the web visit > https://groups.google.com/d/msg/google-appengine/-/z6MbQajkRboJ. > > To post to this group, send email to [email protected]. > To unsubscribe from this group, send email to > [email protected]. > For more options, visit this group at > http://groups.google.com/group/google-appengine?hl=en. -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to [email protected]. To unsubscribe from this group, send email to [email protected]. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
