We have an issue for an asynchronous write API for the datastore: http://code.google.com/p/googleappengine/issues/detail?id=2817
This is something that can fit into that model. On Thu, Feb 25, 2010 at 12:26 PM, Scott Hernandez <scotthernan...@gmail.com>wrote: > Guillermo, > > Taskqueue items can only be 10K > ( > http://code.google.com/appengine/docs/java/taskqueue/overview.html#Quotas_and_Limits > ). > The basic idea is that if you have more data than that you put it into > an entity (in the data-store) and have the task pull it out and > process it. It might be that you can persist those 12K entities in a > lot of large entities (that are still under 1mByte each), but that is > a lot of work for something that will still probably fail. I guess it > all depends where the cost is on the puts (indexing, raw writes by > bytes, number of items). > > And if your comeback is "memcache", well, I won't even start > discussing using a non-persistent, volatile, store like that for > temporary storage while you write them to the datastore... in batches > using the taskqueue/cron/etc. > > Really, there needs to be something that can handle the write volume. > On Thu, Feb 25, 2010 at 12:08 PM, Guillermo Schwarz > <guillermo.schw...@gmail.com> wrote: > > I think there is a way to grab big chunks of oprations, put them in a > > queue to be done asynchronously and that would be it. > > > > My take would be that using proxies it would be easy to queue any long > > operation transparently. I've done that with EJBs in the past, I don't > > see the reason a QueingProxy couldn't be written. > > > > Cheers, > > Guillermo. > > > > On 25 feb, 17:02, Jeff Schnitzer <j...@infohazard.org> wrote: > >> I don't think the original poster had a requirement for synchronous > >> writes; he just didn't want to do the writes asynchronously because it > >> involved a lot more code. > >> > >> I'm also perfectly fine with asynchronous writes and a very lax > >> interpretation of consistency. I don't even mind writing extra code. > >> The thing I worry about is the feasibility of a heavy write load and > >> the total cost of it. > >> > >> Unfortunately I really can't describe in detail what I want to do (I > >> normally laugh at this kind of secrecy, but in this case it's > >> warranted). For the game mechanic I'm thinking about, the > >> average-case scenario is not very far from the worst-case scenario. > >> Just a little detail: > >> > >> * There is no requirement that all of a user's friends must be > >> playing the game or even have installed the app to receive points. > >> Welcome to the world of social gaming, you can play without even > >> without knowing it! > >> * There are *lots* of FB users that have > 1k friends. Probably > >> millions. More active FB users are likely to have more friends... and > >> more likely to use my app. > >> * Points can be assigned to multiple layers, so the # of updates is > >> (layers * friends). > >> * Tens of thousands of people play this "game". It could become > >> hundreds of thousands very soon. If I'm lucky, millions. > >> > >> I would love to implement this game mechanic, but I just can't. > >> Asynchronous or not, it's *way* too expensive on appengine. When it > >> comes time to implement this feature (and it's going to come, I can > >> see the winds blowing), I'm probably going to have to move my scoring > >> system out of appengine. Which is a bit ironic, because one of the > >> main advantages of appengine is scalability. > >> > >> I would *love* to see some sort of super-super-lax and > >> super-super-cheap consistency option for BigTable. Or even an > >> alternative key/value datastore that simply works like a persistent > >> version of memcached. Something that would let me sustain 10k > >> writes/sec without bankrupting me. > >> > >> Jeff > >> > >> On Thu, Feb 25, 2010 at 11:16 AM, Ikai L (Google) <ika...@google.com> > wrote: > >> > >> > Jeff, point taken, but the original poster has been asking for three > >> > different requirements: > >> > - requirement to do all writes synchronously > >> > - sub-some-couple-hundred-millisecond writes > >> > - 12k entities being written > >> > >> > This just won't scale well if it's common. Messaging users can be done > >> > asynchronously, as can the portion crediting friends. I understand the > >> > argument that you may want to do this during the lifecycle of the > request so > >> > the original user gets some kind of feedback backed by a strongly > consistent > >> > datastore, but this just isn't done. Feedback is usually faked out > >> > optimistically, assuming that the writes will all be successful with > some > >> > cached layer being the only part of the stack being updated inside the > >> > request. Thinking of worse case scenarios is a good thought exercise, > but > >> > it's also a bit too optimistic to design a product assuming all of a > Users' > >> > friends will play the game and engineer to meet that unrealistic > >> > expectation. What are the standard and slightly non-standard use > cases? I'd > >> > probably look at a solution where I can store the data somewhere > associated > >> > with the original user for any users not already in the datastore, > then > >> > retrieve and generate a score for any of that User's friends on first > >> > access. Facebook's developer admin tool has some pretty good > statistics such > >> > as bounce rate, block rate and invitation accept rate that can be used > to > >> > tune this design. > >> > Slightly off topic, but we've been asked before if it was possible to > >> > provide different levels of datastore consistency. In some cases I can > see > >> > the tradeoffs making sense. > >> > >> > On Wed, Feb 24, 2010 at 5:52 PM, Jeff Schnitzer <j...@infohazard.org> > wrote: > >> > >> >> On Wed, Feb 24, 2010 at 1:06 PM, Ikai L (Google) <ika...@google.com> > >> >> wrote: > >> >> > My point wasn't necessarily that it wasn't possible. > makePersistentAll > >> >> > does > >> >> > use a batch write, and there are definitely sites that can do > 12,000+ > >> >> > writes > >> >> > a second (and well above that), but I don't know of any that will > >> >> > attempt to > >> >> > do that in a single request. While it's an interesting thought > exercise > >> >> > to > >> >> > see if BigTable can do it through App Engine's interface (hint: it > can, > >> >> > globally, easily), I can't think of a single use case for a site to > need > >> >> > to > >> >> > do this all the time and with the sub-second requirement. I think > it's > >> >> > reasonable to ask why this design exists and why the requirements > exist > >> >> > and > >> >> > rethink one or the other. > >> > >> >> It does seem to be a pretty extreme case, but it's not all that far > >> >> fetched. It's possible for a Facebook user to have 5,000 friends. > >> >> Perhaps a user wants to message all 5k of them. > >> > >> >> I could actually use this right ability now. I would like to add a > >> >> game mechanic which, when you score some points, you also credit a > >> >> portion of that to all of a user's friends. Worst case scenario is a > >> >> 5,000 element read followed by a 5,000 element write. I'm probably > >> >> going to skip this mechanic for now because I can't afford it - even > >> >> with the average 200 or so friends. If I want it badly enough, I may > >> >> ultimately need to move my scoring system offsite. > >> > >> >> Jeff > >> > >> >> -- > >> >> You received this message because you are subscribed to the Google > Groups > >> >> "Google App Engine for Java" group. > >> >> To post to this group, send email to > >> >> google-appengine-j...@googlegroups.com. > >> >> To unsubscribe from this group, send email to > >> >> google-appengine-java+unsubscr...@googlegroups.com<google-appengine-java%2bunsubscr...@googlegroups.com> > . > >> >> For more options, visit this group at > >> >>http://groups.google.com/group/google-appengine-java?hl=en. > >> > >> > -- > >> > Ikai Lan > >> > Developer Programs Engineer, Google App Engine > >> >http://googleappengine.blogspot.com|http://twitter.com/app_engine > >> > >> > -- > >> > You received this message because you are subscribed to the Google > Groups > >> > "Google App Engine for Java" group. > >> > To post to this group, send email to > google-appengine-j...@googlegroups.com. > >> > To unsubscribe from this group, send email to > >> > google-appengine-java+unsubscr...@googlegroups.com<google-appengine-java%2bunsubscr...@googlegroups.com> > . > >> > For more options, visit this group at > >> >http://groups.google.com/group/google-appengine-java?hl=en. > > > > -- > > You received this message because you are subscribed to the Google Groups > "Google App Engine for Java" group. > > To post to this group, send email to > google-appengine-j...@googlegroups.com. > > To unsubscribe from this group, send email to > google-appengine-java+unsubscr...@googlegroups.com<google-appengine-java%2bunsubscr...@googlegroups.com> > . > > For more options, visit this group at > http://groups.google.com/group/google-appengine-java?hl=en. > > > > > > -- > You received this message because you are subscribed to the Google Groups > "Google App Engine for Java" group. > To post to this group, send email to > google-appengine-j...@googlegroups.com. > To unsubscribe from this group, send email to > google-appengine-java+unsubscr...@googlegroups.com<google-appengine-java%2bunsubscr...@googlegroups.com> > . > For more options, visit this group at > http://groups.google.com/group/google-appengine-java?hl=en. > > -- Ikai Lan Developer Programs Engineer, Google App Engine http://googleappengine.blogspot.com | http://twitter.com/app_engine -- You received this message because you are subscribed to the Google Groups "Google App Engine for Java" group. To post to this group, send email to google-appengine-j...@googlegroups.com. To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en.
