On Monday, August 13, 2012 3:10:17 AM UTC-3, Jeff Schnitzer wrote: > > Memcache doesn't work for this application. Completely aside from > reliability issues, there's no memcache instruction for "give me all > the data". At best you can fetch some number of individual keys, but > that brings up two problems:
Reliability comes with costs, as you might know. In some scenarios we've seen, losing few data from millons is not a problem. Maybe that's not your case. > > > 1) Trying to do a batch fetch of 10k keys (or more) > MemCache.getAll(keys) method could work if you know the keys. Using sharded keys you could reach them all. (I get this inspiration from AppStats way of holding an app perf stats ;) 10k keys seems too much. But for 1k, it's working. Why that much? > 2) How do you know what keys to fetch in the first place? > No business meaning keys. For instances numbers from 0 to 999 > > #2 is intractable because any practical solution would essentially > eliminate the need for memcache in the first place. > Well, if you already host your whole app on GAE other 'practical' solution could not exist. > > The problem is pretty easily stated: Collect 10k score submissions in > 5s and be able to provide a sorted leaderboard 5s later. GAE does not > offer any practical facility capable of accomplishing this. > Yeah, that's right :( > > Jeff > > On Fri, Aug 10, 2012 at 1:45 PM, Kristopher Giesing > <[email protected] <javascript:>> wrote: > > Good point. I had unconsciously assumed that a replicated memcached > > solution would be much more efficient than a replicated data store > solution, > > but if the replication and not the disk I/O is what's expensive, then > that > > assumption falls down. > > > > Generally speaking, I'm having difficulty architecting my own app in > such a > > way that it's robust across instances that may be "very distant" from > one > > another. Right now I'm relying on optimistic concurrency in the > datastore, > > but it means my app is much slower than, say, a single instance that > held > > everything in RAM. It's been tempting to just drop the datastore > entirely > > and limit myself to a single instance, but GAE doesn't seem designed to > > scale down that way; front end instances spin up apparently for no > reason, > > and back end instances, as this thread reveals, seem to have more > > performance limitations than one would expect, so I have no idea whether > a > > single instance would actually be able to service my user base. > > > > - Kris > > > > On Friday, August 10, 2012 2:30:35 AM UTC-7, Joakim wrote: > >> > >> By *guaranteed*, do you mean that this should be kept synchronized > across > >> multiple data centers? I ask because in the case of a data center > emergency, > >> your app spins up at a different data center. The HR datastore is > always in > >> sync, the memcache is always local to the DC. > >> Comparing the put-latency of the datastore and memcache for the last > few > >> days, the datastore generally keeps under 50 ms while memcache tends to > stay > >> below 30 ms. I'd say the datastore isn't so bad, and adding sync to > memcache > >> would probably bring that number up a bit. At that point, this new > feature > >> is starting to look a bit like the current HR datastore. Add properties > for > >> insertion date/time etc, and it could be a queue, right? > >> > >> I don't mean to shoot you down, but to make you specify what it really > is > >> you want, and how it relates to the current offering. > >> > > -- > > You received this message because you are subscribed to the Google > Groups > > "Google App Engine" group. > > To view this discussion on the web visit > > https://groups.google.com/d/msg/google-appengine/-/z6MbQajkRboJ. > > > > To post to this group, send email to > > [email protected]<javascript:>. > > > To unsubscribe from this group, send email to > > [email protected] <javascript:>. > > For more options, visit this group at > > http://groups.google.com/group/google-appengine?hl=en. > -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine/-/Pz0QGrbaFT8J. To post to this group, send email to [email protected]. To unsubscribe from this group, send email to [email protected]. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
